the Doppelganger

-
Data scrubbing is a fact of database life. Whenever you import and normalize data you do run the risk of duplicate data imports as well. This can be especially true if your source database had few restrictions on preventing duplicate data entry. In one particular table I had many rows with duplicate data down to the entrytime field; everything was identical except for the PKID. I tried the typical Cursor script found in various websites, but that would Yield an estimate of 14hrs while the test server processed all 16k rows! I could not very well do that to the production server, on my quest I tried several while loops and eventually came up with this select statement which for my situation (2 duplicate rows for 1 real row of data) First thing I did was dump out all the duplicate rows PK's and FK's into a temp table called #Dup, I hit upon this select statment and I sucessfully narrowed down 16k rows down to 8k of unique data, and was able to delete the duplicate FK's based on the PKid.


SELECT A.PKID
FROM #Dup AS A
INNER JOIN (SELECT FKID FROM #Dup GROUP BY FKID) B
ON A.FKID = B.FKID
AND A.PKID IN (SELECT TOP 1 PKID FROM #DUP C WHERE A.FKID = C.FKID)

The Select Statement finds one of the Duplicate tables, you can use this statement to delete only these rows leaving a unique record behind.

Comments

Post a Comment

Popular posts from this blog

Service Broker and External Activator for Sql Server...

-
I ntroduced in Sql Server 2005, Microsoft's Service Broker is a Messaging Queue for Sql Server.  You can audit tables, trigger events, and even call web services (Using the External Activator). Follow this excellent tutorial by Dev Kimchi which helps you step by step and clearly explain how to setup your Service Broker messaging queue. To get started with the external activator you will require the version for the version of Sql Server you are running, click on any of the Microsoft download links and follow the install instructions: Microsoft® SQL Server® 2012 Feature Pack Microsoft® SQL Server® 2014 Feature Pack One last thing before you get frustrated installing the External Activator service on a remote server.  At least for the 2012 version, it still required .Net 3.5 installed on the destination computer/server.  If you are trying to install on a Windows 2012 server follow this helpful guide on how to get .Net 3.5 installed now that the latest packs have been releas
Read more

Lighting Quick Table RowCounts for your Tables

-
Image
If you are familiar with TSQL and want to get a row count from your tables you might be tempted to run a simple Select count(*) from myTable This however can be a lengthy wait if you have millions and millions of records. Not to mention the potential impact on a production system. Instead consider selecting your rowcounts from your system tables SELECT OBJECT_NAME(id), ROWS, indid FROM sysindexes WHERE indid < 2 AND OBJECT_NAME(id) IN ('myTableName') Looking forward beyond Sql Server 2005 and beyond the following select should be used and will provide you with the needed information as system tables access may be depracated SELECT DISTINCT OBJECT_NAME(P.object_id) AS [Name], ROWS FROM sys.indexes I INNER JOIN sys.partitions P ON P.object_id = I.object_id AND P.index_id = I.index_id WHERE I.index_id < 2 AND OBJECT_NAME(P.object_id) IN ('myTableName') So why am I searching for index smaller than 2? In short from the article below you'll find that pure data p
Read more

List to Table

-
E very now and again, a developer will be faced with the task of referincing a List of id's. It's typically irresistable to want to push all these IDs into a single field. If you're searching then you may end up considering a cursor... for that I decided to write the following function... it's like other Tsql functions that push a list to a table, but this is my take on it... I release the code here with out warranty. CREATE FUNCTION udfListToTable (@HList VarChar(2000), @Delimiter CHAR(1)) RETURNS @ListTable TABLE (Field1 VARCHAR(6)) AS BEGIN --By: Francisco Tapia --Date: 2/1/2005 --Purpose: To convert a Comma delimited text to a Temp Variable table To help avoid dynamic sql -- Instead you can join the temp table or use it in your where clause if a field is IN the subquery DECLARE @FieldText as VarChar(6) IF RIGHT(RTRIM(@HLIST),1) <>@Delimiter SET @HList = @HList + @Delimiter WHILE CHARINDEX(@Delimiter, @HList) > 0 BEGIN
Read more