Before answering Keith's specific questions, I'd like to set the stage by describing the fundamentals of Salesforce DX "Projects", "Package Directories", and the "Default Package Directory".
IMPORTANT: If you don't need (or want) to read the "fundamentals" info, just scroll down to the bottom third of this answer. I've got Keith's questions blockquoted down there with the answers just below them.
What is a Salesforce DX "Project"?
In general, a Salesforce DX Project is a new local file structure that collects your org's metadata (code and config), org templates, sample data, and tests. The project root is often the repository root of a version control system (VCS) as well.
Specifically, an SFDX Project exists when you have a local directory that contains the following:
- A project configuration file. This file is always named
sfdx-project.json
and the directory where it's located becomes the "root" of that Salesforce DX project.
- One or more "Salesforce DX Package Directories", which contain SFDX source. These directories live within the SFDX project root, contain SFDX source, and must be explicitly declared as "Package Directories" by adding their path(s) to the
packageDirectories
array inside the project's sfdx-project.json
file.
- A hidden directory named
.sfdx
where the Salesforce CLI keeps a variety of files and directories that support the internal operation of the CLI for that specific Salesforce DX project.
There are a couple more files that you get after running the sfdx force:project:create
command, but the above list describes the minimum set of files and directories that are required to have a functional Salesforce DX project.
What is a Salesforce DX "Package Directory"?
The Salesforce CLI works by scanning all of the Package Directories declared in sfdx-project.json
for locally added or modified SFDX source metadata. It attempts to synchronize that source with any scratch orgs you point the CLI at with sfdx force:source:push
or sfdx force:source:pull
.
Any metadata (Custom Objects, Apex Code, Profiles, etc.) created outside of your SFDX project (like in a scratch org) will be new to the CLI's internal map of your project's metadata. When this happens, the CLI needs to have a place to put newly discovered metadata. This is where the Default Package Directory comes in.
What is the "Default Package Directory" and why is it special?
Defined as part of a project's sfdx-project.json
file, the Default Package Directory is the CLI's go-to location for storing "Remotely Added" metadata.
For example, if you add a new Custom Object named MyObject__c
to your scratch org using the Setup UI and then run sfdx force:source:pull
, the CLI is going to save the SFDX source for your new object locally in your Default Package Directory.
The path to the SFDX source for MyObject__c
will look something like this:
sfdx-project-dir
└─ sfdx-package-dir
└─ main
└─ default
└─ objects
└─ MyObject__c
├─ fields
└─ MyObject__c.object-meta.xml
The names for sfdx-project-dir
and sfdx-package-dir
will be different for your project, but everything else would look exactly like this. The Salesforce CLI will always use the path main/default/<metadata-type>
inside of your default package directory when storing remotely added metadata. This also happens when you use the sfdx force:mdapi:convert
command to convert MDAPI source to SFDX source.
Without a consistent, known location for the CLI to put remotely added SFDX source, developers would need to pre-define locations for all metadata types. Even if a developer thought of "everything" and defined complex rules for what metadata goes where on a force:source:pull
, you'd still need a generic default just in case they missed something.
In other words, having a clean, simple, consistent default location for remotely added metadata is a feature, not a bug. :-)
And now, time to address Keith's original questions (finally!)
Now that we've covered the fundamentals of SFDX Projects and Package Directories, I'll tackle each of Keith's original questions, one at a time.
Question One: Two levels of folder (e.g. "main" and "default") seem
redundant. Why have the "default" level?
I look at main
as a module inside of your Package Directory. This is where the core set of your org customizations would go if you're a customer. If you're building a managed package, main
is where the "shared" code that your app's features depend on lives.
The SFDX-Falcon Template uses this concept of "core" and "feature" metadata quite extensively. The idea is that code/metadata that you put in a feature module can depend on what's in the main module, but not the other way around.
If you're an ISV, logical separation of code/metadata like this doesn't just make it easier to understand and organize your packaged code. It's also a fantastic way to get ready for Second-Generation Packaging (Packaging 2).
So, what about the "redundant" default
directory inside of main?
I don't think it's fair to call the default
directory redundant. It's literally the default location for the CLI to put remotely added metadata. You might implement your own code organization scheme inside of main
, but the CLI is guaranteed to get a known, consistent location to put any new source because it's not interfering with anything "non-default" you might be doing.
Finally, if you're wondering why/how you might further organize SFDX source inside of your main
module, one suggestion is to implement a design pattern based on "separation of concerns". Here's how SFDX-Falcon approaches this:
Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 33)
Question Two: I started out thinking that (for a largish project) I should have > more than one package directory e.g. "force-lts" as well as "force-app" but
in sfdx-project.json only one can be nominated as the default which is
where sfdx force:source:pull pulls changes to. So I'm now thinking it
would be better to just have multiple folders inside the single
"force-app" package directory folder that is marked as or is
implicitly the default so as to avoid a pull dumping files in the
wrong directory. What folder structure works best?
The best answer here depends on whether you're an ISV Partner building a managed package or a customer working on customizations to your production org.
If you're a Customer
If you're a customer with a complex, convoluted, monolithic codebase (aka the "Happy Soup") it can be helpful to try to start breaking things up into many small, independent unmanaged packages wherever possible. Better yet, take a look at Developer Controlled Packages (DCPs) which are part of the Packaging 2 Beta in Spring '18.
Either way, you're using separate SFDX Package Directories with the expectation that you'll want to deploy packages independently of each other as part of a regular agile delivery process. Build fast, build small, deploy often.
If you're an ISV Partner
If you're an ISV building a first-generation managed package, all of your metadata is already in a single package (unless you're using extension packages). When you deploy updates to your packaging org, you're typically going to send everything at once.
The CLI makes it easy to do this because sfdx force:source:convert
operates on only one package directory during the conversion from SFDX to MDAPI source. This makes it easier for ISVs to focus on whole-package deployments when they are ready to push code to their packaging org.
The good news is that you can easily organize source by module inside a single Package Directory anyway. All your core code goes into main
, as I described above. For the rest of your app (ie. your features), you add one or more feature module directories and segment your code, similar to the following.
Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 34)
This is essentially what Keith C is suggesting when he says "it would be better to just have multiple folders inside the single 'force-app' package directory folder". All that SFDX-Falcon does is formalize a basic template for how one might set up those additional folders.
There is one inconvenience that comes up when organizing your SFDX source, however. The fact that doing an sfdx force:source:pull
on remotely added metadata results in the CLI creating SFDX source files in <default-package-dir>/main/default/
.
I personally don't feel like this is a dealbreaker when it comes to choosing to organize SFDX source outside of main/default
. Yes, it's inconvenient to move the source files when you do it but the benefits of having well organized (and hopefully well segmented) code is worth the investment. In fact, the bigger and more complex your code base is, the more you'll benefit over time by investing in getting things organized now.
Conclusion
Salesforce DX brings us new flexibility when organizing the metadata we use to build apps or customize our production orgs. Understanding how the Salesforce CLI uses SFDX Projects and Package Directories to synchronize metadata with scratch orgs can help you get the most out of your Salesforce DX project.
The use of default/main
helps the CLI know where to save new metadata. Organizing your own customizations into modules that are siblings to main
inside of your default Package Directory can make managing large projects easier. Community templates like SFDX-Falcon provide a model for how you can make this happen in your own project, and the investments made now can pay dividends later by having a codebase that's easier to understand and (hopefully!) update.
Read the answers on What's the best practice for putting multiple projects in a git repository? for some possible solutions (over on Stack Overflow). My biggest concern is that if you end up screwing up something in one of the projects, and you try to fix it, you might end up reversing changes in other projects by accident. If you do decide to follow the path of multiple projects in a single repo, consider using "orphan branches", which will help isolate accidental destructive changes. There's details in the answer linked above and also on git-checkout, look for the --orphan
option.
In my personal opinion, you would be better off with separate repositories, because it avoids the possibility that you'll merge something into something else that you didn't mean to do, which can cause an absolute mess of things. While it's "easy" to clean up most mistakes, merging two entire projects together might result in a lot of downstream messes, since you'll basically be forced to backtrack and rebase. Every developer depending on your repo will be forced to undo all their work and clone the repository fresh to avoid any lingering mixups. You don't want this to happen.
Keep in mind, too, that a git repo essentially contains every copy of every change ever committed to the repo when you clone or pull; obviously, this is not literally the case in terms of disk space, but it does make the cloning process longer. This may make clones and pulls more painful than they need to be in terms of processing time. There's other ways of dealing with this, but simply having one repository per project makes that management trivial.
Also, git servers do tend to have an absolute maximum repository size. You'll reach this size sooner if you have multiple, active projects all going at once in the same repository. It might not matter with just Apex Code and Lightning, but if you've got a lot of heavy duty files (data load files, static resources, documents, attachments, etc), you'll exhaust your limits faster than you would with separate repositories.
Note that sfdx uses a project configuration file to pull source. So if you have project1, project2, and project3, and you're in project1's directory, you'll pull the contents of project1 into it, but not project2 or project3. However, if you screw up a username/alias even once, you'll end up having to reset/clean your repository and you'll risk losing work in the scratch org. The last thing you want to do is have a major conflict of (for example) 2,000 files and have to try and manually resolve it.
Best Answer
You need to create some files to make a directory into a SFDX project, particularly
sfdx-project.json
. The easy way to do this is to go to the parent folder of your Git root, and type in the commandsfdx force:project:create -n directoryName
. This creates a number of files from a template that will be used to configure the project. After this, you should then convert your source folder using the steps below.That looks like this:
You may also see "conflict" and "force"; this means files were overwritten. You can
git checkout
to revert those changes afterwards.Git is good about detecting the file name changes. However, it has a built-in limit on a number of files it can handle at a time.
Hence when you convert your project to sfdx format chances of losing history are high since number of files that get renamed in this process may be higher than the limit that git has.
You can increase this limit by a simple config like below
Here is what you will need to preserve the history