the Doppelganger

-
Data scrubbing is a fact of database life. Whenever you import and normalize data you do run the risk of duplicate data imports as well. This can be especially true if your source database had few restrictions on preventing duplicate data entry. In one particular table I had many rows with duplicate data down to the entrytime field; everything was identical except for the PKID. I tried the typical Cursor script found in various websites, but that would Yield an estimate of 14hrs while the test server processed all 16k rows! I could not very well do that to the production server, on my quest I tried several while loops and eventually came up with this select statement which for my situation (2 duplicate rows for 1 real row of data) First thing I did was dump out all the duplicate rows PK's and FK's into a temp table called #Dup, I hit upon this select statment and I sucessfully narrowed down 16k rows down to 8k of unique data, and was able to delete the duplicate FK's based on the PKid.


SELECT A.PKID
FROM #Dup AS A
INNER JOIN (SELECT FKID FROM #Dup GROUP BY FKID) B
ON A.FKID = B.FKID
AND A.PKID IN (SELECT TOP 1 PKID FROM #DUP C WHERE A.FKID = C.FKID)

The Select Statement finds one of the Duplicate tables, you can use this statement to delete only these rows leaving a unique record behind.

Comments

Post a Comment

Popular posts from this blog

Service Broker and External Activator for Sql Server...

-
I ntroduced in Sql Server 2005, Microsoft's Service Broker is a Messaging Queue for Sql Server.  You can audit tables, trigger events, and even call web services (Using the External Activator). Follow this excellent tutorial by Dev Kimchi which helps you step by step and clearly explain how to setup your Service Broker messaging queue. To get started with the external activator you will require the version for the version of Sql Server you are running, click on any of the Microsoft download links and follow the install instructions: Microsoft® SQL Server® 2012 Feature Pack Microsoft® SQL Server® 2014 Feature Pack One last thing before you get frustrated installing the External Activator service on a remote server.  At least for the 2012 version, it still required .Net 3.5 installed on the destination computer/server.  If you are trying to install on a Windows 2012 server follow this helpful guide on how to get .Net 3.5 installed now that the latest packs have b...
Read more

List to Table

-
E very now and again, a developer will be faced with the task of referincing a List of id's. It's typically irresistable to want to push all these IDs into a single field. If you're searching then you may end up considering a cursor... for that I decided to write the following function... it's like other Tsql functions that push a list to a table, but this is my take on it... I release the code here with out warranty. CREATE FUNCTION udfListToTable (@HList VarChar(2000), @Delimiter CHAR(1)) RETURNS @ListTable TABLE (Field1 VARCHAR(6)) AS BEGIN --By: Francisco Tapia --Date: 2/1/2005 --Purpose: To convert a Comma delimited text to a Temp Variable table To help avoid dynamic sql -- Instead you can join the temp table or use it in your where clause if a field is IN the subquery DECLARE @FieldText as VarChar(6) IF RIGHT(RTRIM(@HLIST),1) @Delimiter SET @HList = @HList + @Delimiter WHILE CHARINDEX(@Delimiter, @HList) > 0 BEGIN ...
Read more

Turbo Charge Your SSMS

-
The other day a co-worker mentioned one of his biggest pet peeves about Sql's new IDE. Of course he likes the old IDE, but what he detested the most was the slow bloat time it took to get his SSMS loaded on the screen. When we timed the boot time for his Enterprise Manager to load it took nearly 7 seconds. Not bad, but then proceeded to tell me how much longer SSMS (Sql Server Management Studio) took to load. I guessed that it could possibly be about 10 seconds (only 3 more seconds) but I was shocked to see it load in nearly 25 seconds. it was nearly 3x less productive just loading the thing before you could connect to do anything. I couldn't believe that it was nearly that slow on his computer, I had never timed it on my pc. So when I returned I had to try it. Again the load time was nearly 20 seconds on my machine (about 18 seconds to be exact). I was not entirely pleased with my results but thankfully Google provided the answer, one quick search for "Turbo char...
Read more