[SalesForce] Setup Salesforce(Sandbox) project in Visual Studio Code and integrate source control with Bitbucket

I am new to VS Code, things I was able to do separately are

Create a Salesforce Manifest project and authorize the org, Retrieve/Deploy are good.
Have a repository on Bitbucket, cloned it on the VS Code, push/pull operations are good.

Problem is I am not able to make it a continuous process, retrieve from Sandbox and push it to Bitbucket, vice versa. Below are the steps followed in setup. Please help where I am going wrong.

SFDX: Create Project with Manifest on VS Code, named Demo-Project under ../Documents folder.
SFDX: Authorize an Org, credentials entered, success. Package xml under manifest folder. Empty aura, lwc folders under ../Documents/Demo-Project/force-app/main/default
Under Bitbucket, I have repository Remote-Project, this has source classes, pages, etc. I copied the clone URL.
GIT: Clone on VS Code. Directory selected ../Documents/Demo-Project/force-app. A folder named remote-project is created with the source content from bitbucket. Got a pop up asking for Open Repository and Add to Workspace. I clicked on Add to Workspace.
Opened a file from ../Documents/Demo-Project/force-app/remote-project/aura/democmp, made changes, right click, Deploy to Source org, it got deployed. Source Control tracks the changes.
Now, when I Retrieve from Source Org, a new file named democmp is created under ../Documents/Demo-Project/force-app/main/default/aura, and the Source Control does not track this changes.

I understand I am doing wrong at some setup step, please help me here to make this a continuous process.

Best Answer

GIT: Clone on VS Code. Directory selected ../Documents/Demo-Project/force-app. A folder named remote-project is created with the source content from bitbucket. Got a pop up asking for Open Repository and Add to Workspace. I clicked on Add to Workspace.

This is your issue. You're creating an SFDX project, and then cloning a different SFDX project inside it. When you complete this step, your directory tree probably looks like this:

Demo-Project/
    force-app/
        main/
            default/
                aura/
        remote-project/
            .git/
            aura/
                democmp/

Only the folder enclosing .git and its descendants are part of the Git repository, because that's where you chose to clone it. Git cannot see anything outside the repo boundary, so anything that is saved in main/default/aura/ isn't part of the repo and isn't visible to Git.

Conversely, you've told Visual Studio Code, by creating an SFDX project, that you want to be working in an outer directory tree rooted at force-app. If you look in your sfdx-project.json file, you should see something like

"packageDirectories": [
  {
    "path": "force-app",
    "default": true
  }
],

This tells VSC that your project's primary source directory is under force-app (not remote-project), and that's where your SFDX commands are going to save and push source from by default.

If you're starting from an existing SFDX project that's stored in a source code repo, which sounds like it's the case although I suspect something's a little off about the way you're discussing the folder tree, you probably want to just clone that project and then open its root folder in Visual Studio Code, rather than creating a new project locally first.

What is a Salesforce DX "Project"?

In general, a Salesforce DX Project is a new local file structure that collects your org's metadata (code and config), org templates, sample data, and tests. The project root is often the repository root of a version control system (VCS) as well.

Specifically, an SFDX Project exists when you have a local directory that contains the following:

A project configuration file. This file is always named sfdx-project.json and the directory where it's located becomes the "root" of that Salesforce DX project.
One or more "Salesforce DX Package Directories", which contain SFDX source. These directories live within the SFDX project root, contain SFDX source, and must be explicitly declared as "Package Directories" by adding their path(s) to the packageDirectories array inside the project's sfdx-project.json file.
A hidden directory named .sfdx where the Salesforce CLI keeps a variety of files and directories that support the internal operation of the CLI for that specific Salesforce DX project.

There are a couple more files that you get after running the sfdx force:project:create command, but the above list describes the minimum set of files and directories that are required to have a functional Salesforce DX project.

What is a Salesforce DX "Package Directory"?

The Salesforce CLI works by scanning all of the Package Directories declared in sfdx-project.json for locally added or modified SFDX source metadata. It attempts to synchronize that source with any scratch orgs you point the CLI at with sfdx force:source:push or sfdx force:source:pull.

Any metadata (Custom Objects, Apex Code, Profiles, etc.) created outside of your SFDX project (like in a scratch org) will be new to the CLI's internal map of your project's metadata. When this happens, the CLI needs to have a place to put newly discovered metadata. This is where the Default Package Directory comes in.

What is the "Default Package Directory" and why is it special?

Defined as part of a project's sfdx-project.json file, the Default Package Directory is the CLI's go-to location for storing "Remotely Added" metadata.

For example, if you add a new Custom Object named MyObject__c to your scratch org using the Setup UI and then run sfdx force:source:pull, the CLI is going to save the SFDX source for your new object locally in your Default Package Directory.

The path to the SFDX source for MyObject__c will look something like this:

sfdx-project-dir
└─ sfdx-package-dir
   └─ main
      └─ default
         └─ objects
            └─ MyObject__c
               ├─ fields
               └─ MyObject__c.object-meta.xml

The names for sfdx-project-dir and sfdx-package-dir will be different for your project, but everything else would look exactly like this. The Salesforce CLI will always use the path main/default/<metadata-type> inside of your default package directory when storing remotely added metadata. This also happens when you use the sfdx force:mdapi:convert command to convert MDAPI source to SFDX source.

Without a consistent, known location for the CLI to put remotely added SFDX source, developers would need to pre-define locations for all metadata types. Even if a developer thought of "everything" and defined complex rules for what metadata goes where on a force:source:pull, you'd still need a generic default just in case they missed something.

In other words, having a clean, simple, consistent default location for remotely added metadata is a feature, not a bug. :-)

And now, time to address Keith's original questions (finally!)

Now that we've covered the fundamentals of SFDX Projects and Package Directories, I'll tackle each of Keith's original questions, one at a time.

Question One: Two levels of folder (e.g. "main" and "default") seem redundant. Why have the "default" level?

I look at main as a module inside of your Package Directory. This is where the core set of your org customizations would go if you're a customer. If you're building a managed package, main is where the "shared" code that your app's features depend on lives.

The SFDX-Falcon Template uses this concept of "core" and "feature" metadata quite extensively. The idea is that code/metadata that you put in a feature module can depend on what's in the main module, but not the other way around.

If you're an ISV, logical separation of code/metadata like this doesn't just make it easier to understand and organize your packaged code. It's also a fantastic way to get ready for Second-Generation Packaging (Packaging 2).

So, what about the "redundant" default directory inside of main?

I don't think it's fair to call the default directory redundant. It's literally the default location for the CLI to put remotely added metadata. You might implement your own code organization scheme inside of main, but the CLI is guaranteed to get a known, consistent location to put any new source because it's not interfering with anything "non-default" you might be doing.

Finally, if you're wondering why/how you might further organize SFDX source inside of your main module, one suggestion is to implement a design pattern based on "separation of concerns". Here's how SFDX-Falcon approaches this:

Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 33)

Question Two: I started out thinking that (for a largish project) I should have > more than one package directory e.g. "force-lts" as well as "force-app" but in sfdx-project.json only one can be nominated as the default which is where sfdx force:source:pull pulls changes to. So I'm now thinking it would be better to just have multiple folders inside the single "force-app" package directory folder that is marked as or is implicitly the default so as to avoid a pull dumping files in the wrong directory. What folder structure works best?

The best answer here depends on whether you're an ISV Partner building a managed package or a customer working on customizations to your production org.

If you're a Customer

If you're a customer with a complex, convoluted, monolithic codebase (aka the "Happy Soup") it can be helpful to try to start breaking things up into many small, independent unmanaged packages wherever possible. Better yet, take a look at Developer Controlled Packages (DCPs) which are part of the Packaging 2 Beta in Spring '18.

Either way, you're using separate SFDX Package Directories with the expectation that you'll want to deploy packages independently of each other as part of a regular agile delivery process. Build fast, build small, deploy often.

If you're an ISV Partner

If you're an ISV building a first-generation managed package, all of your metadata is already in a single package (unless you're using extension packages). When you deploy updates to your packaging org, you're typically going to send everything at once.

The CLI makes it easy to do this because sfdx force:source:convert operates on only one package directory during the conversion from SFDX to MDAPI source. This makes it easier for ISVs to focus on whole-package deployments when they are ready to push code to their packaging org.

The good news is that you can easily organize source by module inside a single Package Directory anyway. All your core code goes into main, as I described above. For the rest of your app (ie. your features), you add one or more feature module directories and segment your code, similar to the following.

Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 34)

This is essentially what Keith C is suggesting when he says "it would be better to just have multiple folders inside the single 'force-app' package directory folder". All that SFDX-Falcon does is formalize a basic template for how one might set up those additional folders.

There is one inconvenience that comes up when organizing your SFDX source, however. The fact that doing an sfdx force:source:pull on remotely added metadata results in the CLI creating SFDX source files in <default-package-dir>/main/default/.

I personally don't feel like this is a dealbreaker when it comes to choosing to organize SFDX source outside of main/default. Yes, it's inconvenient to move the source files when you do it but the benefits of having well organized (and hopefully well segmented) code is worth the investment. In fact, the bigger and more complex your code base is, the more you'll benefit over time by investing in getting things organized now.

Conclusion

Salesforce DX brings us new flexibility when organizing the metadata we use to build apps or customize our production orgs. Understanding how the Salesforce CLI uses SFDX Projects and Package Directories to synchronize metadata with scratch orgs can help you get the most out of your Salesforce DX project.

The use of default/main helps the CLI know where to save new metadata. Organizing your own customizations into modules that are siblings to main inside of your default Package Directory can make managing large projects easier. Community templates like SFDX-Falcon provide a model for how you can make this happen in your own project, and the investments made now can pay dividends later by having a codebase that's easier to understand and (hopefully!) update.

[SalesForce] How to push code to production with Visual Studio Code & DX

You have two choices. First, you can use force:package commands to create an unlocked package. You can install this in production with force:package:install; it is similar to installing an AppExchange package. Otherwise, you would need to use force:source:convert to get a mdapi style format, then use force:mdapi:deploy with the relevant test level (e.g. RunLocalTests) and set rollbackOnError to true. Check out the Salesforce CLI Reference for specific details, or sfdx help force:<command> for command-level help direct from the console. I do not believe that you're currently allowed to create unlocked packages directly via the GUI, but you can open a terminal by using the Terminal > Create New Terminal option, which will open a sfdx environment by default.