Duplication issue

[pavlo888]

Hi @ThWacker

I am interested in using this package for designing some specific primers for fungal targets. However, I am having issues with the script. I am sharing the log.

For context, this is the command I am trying to run,
bash ./DiPPER2/scripts/DiPPER2_wrapper.sh -d /home/usuari/Descargas/Aureo-primers/ -f /home/usuari/Descargas/Aureo-primers/Aureo-primers-test -o Aureo -l /home/usuari/Descargas/Aureo-primers/target-aureo.txt

And this is the list of genomes included:
Aureobasidium_melanogenum.fna
Aureobasidium_pullulans.fna
Hortaea_werneckii.fna

I am also including the list of target genomes as a .txt file.

Could you please let me know how to proceed in order to get the script running?

target-aureo.txt

application.log

Cheers,
Pablo

[ThWacker]

Hi @pavlo888 ,

thank you very much for being interested in my package and thank you for sending me the info regarding your issue!
From the application log, it appears like the targets_move module does not receive the full path, but a relative path for some reason. Can you let me know if this application.log was generated from exactly the command you pasted above you tried to run? Your pasted command above shows an absolute path, which technically should be ok. Additionally, I would put the assemblies in a subfolder of Aureo-primers ideally and make sure that there is only one assembly per accession. Assemblies get upversioned sometimes (GCF_1234567.1 and GCF_1234567.2 for instance) and only one of both should be present, ideally the one of higher quality. What is your folder structure overall?

Many thanks and best wishes,
Theresa

[davidjstudholme]

@ThWacker, when DiPPER2 is checking for duplicate accession numbers, where does it look for those accession numbers? I mean, is it the IDs in the FASTA headings that are being treated as accession numbers? Or is it assuming that the filenames are accession numbers? I think it is unsafe to assume that the IDs and/or filenames are accession numbers rather than arbitrary names.

[davidjstudholme]

@ThWacker, I am experiencing the same issue. I am getting "ERROR - target_move_module_optimized.py - Make sure that there is only one assembly per accession. Please delete the other assemblies and only keep the highest quality one."

The output does not state which "accession" isduplicated.

Here is my command line:

./DiPPER2/scripts/DiPPER2_wrapper.sh -d /data/djs217/PROJECTS/45_ralstonia/07_dipper2/genomes -f /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow -o UW551-narrow -q n -l /data/djs217/PROJECTS/45_ralstonia/07_dipper2/targets.UW551-narrow.txt

Here is the output:

...
Requirement already satisfied: platformdirs>=2.2 in ./dipper2/lib/python3.12/site-packages (from pylint->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 5)) (4.9.6)
Requirement already satisfied: tomlkit>=0.10.1 in ./dipper2/lib/python3.12/site-packages (from pylint->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 5)) (0.15.0)
Requirement already satisfied: six>=1.5 in ./dipper2/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 4)) (1.17.0)
Requirement already satisfied: biothings-client>=0.2.6 in ./dipper2/lib/python3.12/site-packages (from mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.5.0)
Requirement already satisfied: httpx>=0.25.0 in ./dipper2/lib/python3.12/site-packages (from biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.28.1)
Requirement already satisfied: anyio in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (4.13.0)
Requirement already satisfied: certifi in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (2026.4.22)
Requirement already satisfied: httpcore==1.* in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (1.0.9)
Requirement already satisfied: idna in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (3.15)
Requirement already satisfied: h11>=0.16 in ./dipper2/lib/python3.12/site-packages (from httpcore==1.*->httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.16.0)
Requirement already satisfied: typing_extensions>=4.5 in ./dipper2/lib/python3.12/site-packages (from anyio->httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (4.15.0)
Requirement already satisfied: packaging>=20.0 in ./dipper2/lib/python3.12/site-packages (from pooch->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (26.2)
Requirement already satisfied: charset_normalizer<4,>=2 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (3.4.7)
Requirement already satisfied: urllib3<3,>=1.26 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (2.7.0)
Shifting all the target assemblies into FUR.target and all the neighbours into FUR.neighbour folders
2026-05-14 20:01:05,359 - ERROR - target_move_module_optimized.py - Make sure that there is only one assembly per accession. Please delete the other assemblies and only keep the highest quality one.
Running FUR
2026-05-14 20:01:05,439 - ERROR - FUR_module_optimized.py - The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.neighbour is empty
The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.neighbour is empty
FUR failed with error 1. Check log for details!

Here is my directory structure:

tree > directory-structure.txt

directory-structure.txt

[ThWacker]

@davidjstudholme I am investigating this issue, but just to see whether this might be a source of the problem: can you remove the "config" from each of the file names? This should not be an issue, but it narrows down the source of the bug to see what happens when you do. Thank you!

[davidjstudholme]

Thank you, @ThWacker. I appreciate your looking into this in your own time.

I tried renaming the files and running again, but the same problem persists.

Here is my command line:

./DiPPER2/scripts/DiPPER2_wrapper.sh -d /data/djs217/PROJECTS/45_ralstonia/07_dipper2/genomes -f /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow -o UW551-narrow -q n -l /data/djs217/PROJECTS/45_ralstonia/07_dipper2/targets.UW551-narrow.txt

Here is the output:

...
Requirement already satisfied: platformdirs>=2.2 in ./dipper2/lib/python3.12/site-packages (from pylint->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 5)) (4.9.6)
Requirement already satisfied: tomlkit>=0.10.1 in ./dipper2/lib/python3.12/site-packages (from pylint->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 5)) (0.15.0)
Requirement already satisfied: six>=1.5 in ./dipper2/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 4)) (1.17.0)
Requirement already satisfied: biothings-client>=0.2.6 in ./dipper2/lib/python3.12/site-packages (from mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.5.0)
Requirement already satisfied: httpx>=0.25.0 in ./dipper2/lib/python3.12/site-packages (from biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.28.1)
Requirement already satisfied: anyio in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (4.13.0)
Requirement already satisfied: certifi in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (2026.4.22)
Requirement already satisfied: httpcore==1.* in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (1.0.9)
Requirement already satisfied: idna in ./dipper2/lib/python3.12/site-packages (from httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (3.15)
Requirement already satisfied: h11>=0.16 in ./dipper2/lib/python3.12/site-packages (from httpcore==1.*->httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (0.16.0)
Requirement already satisfied: typing_extensions>=4.5 in ./dipper2/lib/python3.12/site-packages (from anyio->httpx>=0.25.0->biothings-client>=0.2.6->mygene->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (4.15.0)
Requirement already satisfied: packaging>=20.0 in ./dipper2/lib/python3.12/site-packages (from pooch->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (26.2)
Requirement already satisfied: charset_normalizer<4,>=2 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (3.4.7)
Requirement already satisfied: urllib3<3,>=1.26 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (2.7.0)
Shifting all the target assemblies into FUR.target and all the neighbours into FUR.neighbour folders
2026-05-16 11:14:11,299 - ERROR - target_move_module_optimized.py - Make sure that there is only one assembly per accession. Please delete the other assemblies and only keep the highest quality one.
Running FUR
2026-05-16 11:14:11,383 - ERROR - FUR_module_optimized.py - The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.neighbour is empty
The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.neighbour is empty
FUR failed with error 1. Check log for details!

Note: "2026-05-16 11:14:11,299 - ERROR - target_move_module_optimized.py - Make sure that there is only one assembly per accession. Please delete the other assemblies and only keep the highest quality one."

Here are my filenames:

ls genomes/

068_2.fna        EN1.fna         LMG_9673.fna     MAFF_106256.fna  MAFF_301524.fna  MAFF_311888.fna  MAFF_724001.fna     Po82.fna     RUN6499.fna  UACh012.fna
071-2.fna        EP1.fna         LcA12172.fna     MAFF_106257.fna  MAFF_311771.fna  MAFF_311889.fna  MAFF_724002.fna     R24.fna      RUN6534.fna  UGMSLYSS_Db01.fna
077-080_1.fna    FQY_4.fna       MAFF_106204.fna  MAFF_106258.fna  MAFF_311772.fna  MAFF_311890.fna  MAFF_724003.fna     RS.fna       RalTom1.fna  UGMSS_Db01.fna
095-3.fna        GBBC_1122.fna   MAFF_106205.fna  MAFF_106259.fna  MAFF_311800.fna  MAFF_311891.fna  MAFF_724004.fna     RS2.fna      Rs5.fna      UMN24-1.fna
102-3.fna        GBBC_1172.fna   MAFF_106206.fna  MAFF_106260.fna  MAFF_311855.fna  MAFF_311892.fna  MAFF_724005.fna     RS26.fna     SL1931.fna   UMN24-2.fna
115-4.fna        GBBC_1200.fna   MAFF_106207.fna  MAFF_106261.fna  MAFF_311869.fna  MAFF_311893.fna  MAFF_724008.fna     RS638.fna    SL2064.fna   UW163.fna
144-1.fna        GBBC_1202.fna   MAFF_106208.fna  MAFF_106263.fna  MAFF_311870.fna  MAFF_311894.fna  MAFF_724009.fna     RS639.fna    SL2312.fna   UW251.fna
19-3PR_UW348.fa  GBBC_1331.fna   MAFF_106209.fna  MAFF_106264.fna  MAFF_311871.fna  MAFF_311895.fna  MAFF_724010.fna     RS642.fna    SL3022.fna   UW386.fna
A2-HR_MARDI.fna  GBBC_1338.fna   MAFF_106210.fna  MAFF_106265.fna  MAFF_311872.fna  MAFF_311896.fna  MLY158.fna          RS647.fna    SL3175.fna   UW559_UCD729.fna
AGQB-112.fna     GBBC_1341.fna   MAFF_106211.fna  MAFF_106266.fna  MAFF_311873.fna  MAFF_311897.fna  MSG-1.fna           RS650.fna    Sw698.fna    UW602_UCD737.fna
AHR56.fna        GBBC_1956.fna   MAFF_106212.fna  MAFF_106267.fna  MAFF_311874.fna  MAFF_311898.fna  MolK2.fna           RS848.fna    T101.fna     UW605_UCD733.fna
BA7.fna          GBBC_3080.fna   MAFF_106213.fna  MAFF_106268.fna  MAFF_311875.fna  MAFF_311899.fna  NCPPB_3725.fna      RS_476.fna   T11.fna      UW606_UCD721.fna
BDBR229.fna      GMI1000.fna     MAFF_106214.fna  MAFF_106269.fna  MAFF_311876.fna  MAFF_311900.fna  NCPPB_3727.fna      RS_488.fna   T12.fna      UW607_UCD722.fna
Br19.fna         Gazipur_4.fna   MAFF_106215.fna  MAFF_107632.fna  MAFF_311877.fna  MAFF_311901.fna  NCPPB_3728.fna      RS_489.fna   T51.fna      UW608_UCD723.fna
CCRMRsB7.fa      Gazipur_5.fna   MAFF_106216.fna  MAFF_211266.fna  MAFF_311878.fna  MAFF_311902.fna  NCSU_Rs32.fna       RUN0320.fna  T523.fna     UW609_UCD734.fna
CFBP2957.fna     Gj707.fna       MAFF_106217.fna  MAFF_211267.fna  MAFF_311879.fna  MAFF_311906.fna  NCSU_Rs45488.fna    RUN2161.fna  T82.fna      UW72.fna
CFBP_8695.fna    IBSBF1503.fna   MAFF_106218.fna  MAFF_211268.fna  MAFF_311880.fna  MAFF_311907.fna  NCSU_RsMR-FL01.fna  RUN2279.fna  T95.fna      UW763.fna
CFBP_8697.fna    IBSBF_2571.fna  MAFF_106244.fna  MAFF_211269.fna  MAFF_311881.fna  MAFF_311908.fna  NIB_Z_3685.fna      RUN2340.fna  T98.fna      UY031.fna
CFIA906.fna      INTABV18.fna    MAFF_106245.fna  MAFF_211271.fna  MAFF_311882.fna  MAFF_311909.fna  P597.fna            RUN2474.fna  UA-1579.fna  XCqk.fna
CIAT_078.fna     INTABV2657.fna  MAFF_106251.fna  MAFF_211282.fna  MAFF_311883.fna  MAFF_311910.fna  P673.fna            RUN3149.fna  UA-1591.fna  XZqk.fna
CIP417_UW70.fna  INTABV29.fna    MAFF_106252.fna  MAFF_211519.fna  MAFF_311884.fna  MAFF_311931.fna  PD_7123.fna         RUN3223.fna  UA-1609.fna  YLqk.fna
CQPS-1.fna       INTABV624.fna   MAFF_106253.fna  MAFF_211520.fna  MAFF_311885.fna  MAFF_331041.fna  PSI07.fna           RUN5935.fna  UA-1611.fna  application.log
Cw717.fna        KACC_10722.fna  MAFF_106254.fna  MAFF_211521.fna  MAFF_311886.fna  MAFF_331042.fna  PSS4.fna            RUN6285.fna  UA-1612.fna  rename_files.sh
DG-3.fna         LLRS-1.fna      MAFF_106255.fna  MAFF_301069.fna  MAFF_311887.fna  MAFF_331046.fna  PeaFJ1.fna          RUN6340.fna  UA-1617.fna  uw551.fa

Here is my targets list:

cat targets.UW551-narrow.txt

CFBP_8695.fna
CFIA906.fna
RS2.fna
RS_488.fna
UACh012.fna
UW72.fna
UY031.fna
uw551.fa

I can send you the genomes FASTA file privately if that helps you to replicate the error.

[ThWacker]

@davidjstudholme That would be great if you could send me those files! Thank you!

[ThWacker]

@davidjstudholme @pavlo888 Figured out the reason for the bug, and will push a fix the next couple days. DiPPER2 was originally meant to be used with GCA/ GCF accessions only. For those, it works robustly, as they are very uniform in their pattern. However, when "random" IDs are used for assemblies, it can choke on a specific regex issue. I will fix it asap and let you know when I have.

[davidjstudholme]

Thank you, @ThWacker. That makes sense. That explains why we refer to "accessions" rather than filenames in the preprint. I think this current behaviour is fines, so long as the README clearly states that it is only intended for use with GenBank/RefSeq genome assemblies. However, it would be a nice feature to be able to include "random" genome sequences too.

[ThWacker]

@davidjstudholme could you kindly checkout your dipper repo to the feature branch feature/accession_processing_target_moving_branch after pulling and tell me if the refactored target_move_module_optimized.py script there solves the problem? Many thanks in advance!

[davidjstudholme]

@ThWacker, my job, which previously failed, has now been running for some time without error. So, this might be resolved now. I will let you know when/if the job finishes.

[davidjstudholme]

@ThWacker, now it fails with a different error message. So, probably this issue is resolved but there may or may not be another issue. I will need to test it further.

Output:

...
data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (4.15.0)
Requirement already satisfied: packaging>=20.0 in ./dipper2/lib/python3.12/site-packages (from pooch->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (26.2)
Requirement already satisfied: charset_normalizer<4,>=2 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (3.4.7)
Requirement already satisfied: urllib3<3,>=1.26 in ./dipper2/lib/python3.12/site-packages (from requests->Bio->-r /data/djs217/PROJECTS/45_ralstonia/07_dipper2/DiPPER2/requirements.txt (line 1)) (2.7.0)
Shifting all the target assemblies into FUR.target and all the neighbours into FUR.neighbour folders
Running FUR
using RS2.fna as target representative



making Blast database


2026-05-29 19:17:11,938 - ERROR - FUR_module_optimized.py - Fur could not find unique regions or did not run successfully. /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/UW551-narrow_FUR.db.out.txt is empty or does not exist.
Finished running FUR
Pick primers using Primer3
2026-05-29 19:17:12,226 - ERROR - Primer3_module_optimized.py - /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/UW551-narrow_FUR.db.out.primers.txt is empty.
Finished primer picking.
Testing the primers for specificity and sensitivity in silico and determining the target
2026-05-29 19:17:12,527 - ERROR - Primer_Testing_module_optimized.py - The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.P3.PRIMERS does not exist
The folder /data/djs217/PROJECTS/45_ralstonia/07_dipper2/out.UW551-narrow/FUR.P3.PRIMERS does not exist
Primer testing failed with error 1. Check log for details!

[ThWacker]

@davidjstudholme is the above issue the same as the one in the issue #12? If so, as with the issue #12: the problem here is that FUR did only find one unique region and could not pick primers for that. This is an issue with the input sequences and/ or phylogeny in that case, and not with DiPPER2. However, I should make the logging and error messages clearer in that aspect, so that people understand that this is not a pipeline error.