Categories
Linux

How an ADO pipeline can modify its own repo

Intro

I wanted to run a job on an Azure DevOps pipeline which did a backup of DNS zones on Cloudflare and write the results, in the form of a compressed tar file, into the ADO repository since everyone on the team has access to it and knows how to make a clone of the repo.

My first attempts produced some stunningly bad results. I was wiping out recently created files in the repo and such. That is very undesirable.

The solution

By “stealing with pride” from colleagues and such, I arrived at this AFAIK working solution. Here is the yaml file.

trigger: none

pool:
  name: backup_agents

steps:
# next two lines needed so we can modify the git repo and add our backups
- checkout: self
  clean: true
  persistCredentials: true
  fetchDepth: 1

- script: pip3 install -vvv --timeout 60 -r Cloudflare-backup/requirements.txt
  displayName: 'Install requirements'

- script: python3 backup-all-zones.py
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/Cloudflare-backup
  env:
    CLOUDFLARE_API_TOKEN: $(cloudflare_api_token)
    PYTHONPATH: $(System.DefaultWorkingDirectory)/Cloudflare-backup:$(System.DefaultWorkingDirectory)
- script: |
    git config --global http.sslVerify false
    git config --global user.email "[email protected]"
    git config --global user.name "pipeline"
    cd Cloudflare-backup
    pwd
    ls
    git add backups/zones-*
    git commit -m "adding todays backup files"
    git push origin HEAD:refs/heads/main

schedules:
- cron: "47 23 * * *"
  displayName: Run the script at 23:47 UTC
  branches:
    include:
    - main

I’m not exactly where all the magic happens. I think the section at the top that does the self checkout must be important. Then, obviously, there are the git add/git commit -m/git push commands. I do not claim to understand the origin HEAD:refs/heads/main argument to git push. I just copied it from a working example.

And branches: include -main. I’m not sure what this does either.

I need a few more days of testing, to be really certain, but I no longer am reverting my repo to an old state as I was with my initial attempts which involved doing a git fetch and probably missed the self checkout step as well.

Conclusion

One day I hope to understand git. But that today is not today! nevertheless I got my ADO pipeline to add backup files to its own ADO repository! So that’s cool.

References and related

My own git cheatsheet

Cloudflare python api examples

Categories
Admin

Git commands cheat sheet

Intro

This is the list of git commands I compiled.

Create new local GIT repository

git init [project name]

Copy a repository

git clone username@host:/path/to/repository

Add a file to the staging area – must be done or won’t be saved in next commit

git add temp.txt or git add -A (add everything at once)

My working style is to change one to three files, and only add those.

Suppose you run git add, then change the file again, and then run git commit: what version will you push? The original! Run git add, etc all over again to pick up the latest changes.

Create a snapshot of the changes and save to git directory??

Note that any committed changes won’t make their way to the remote repo

git commit –m “Message to go with the commit here”

Nota bene: git does not allow to add empty folders! If you try, you’ll simply see: nothing to commit, working tree clean

Put some “junk file” like .gitkeep in your empty folder for git add/git commit to work.

Set user-specific values

git config –global user.email youremail

Displays the list of changed files together with the files that are yet to be staged or committed

git status

Undo that git add you just did

git status to see what’s going on. Then

git restore –staged filename

and then another git status to see that that worked.

Send local commits to the master branch of the remote repository

  (Replace <master> with the branch where you want to push your changes when you’re not intending to push to the master branch)

git push origin <master>

For real basic setups like mine, where I work on branch master, it suffices to simply do git push

Merge all the changes present in the remote repository to the local working directory

git pull

Create branches and helps you to navigate between them

git checkout -b <branch-name>

Switch from one branch to another

git checkout <branch-name>

View all remote repositories

git remote -v

Connect the local repository to a remote server

git remote add origin <host-or-remoteURL>

Delete connection to a specified remote repository

git remote rm <name-of-the-repository>

List, create, or delete branches

git branch

Delete a branch

git -d <branch-name>

Merge a branch into the active one

git merge <branch-name>

List file conflicts

git diff –base <file-name>

View conflicts between branches before a merge

git diff <source-branch> <target-branch>

List all conflicts

git diff

Mark certain commits, i.e., v1.0

git tag <commitID>

View repository’s commit history, etc

git log, e.g., git log –oneline

Reset index?? and working directory to last git commit

git reset –hard HEAD

Remove files from the index?? and working directory

git rm filename.txt

Revert (undo) changes from a commit as per hash shown by git log –oneline

git revert <hash>

Temporarily save the changes not ready to be committed??

git stash

View info about any git object

git show

Fetch all objects from the remote repository that don’t currently reside in the local working directory

git fetch origin

View a tree object??

git ls-tree HEAD

Search everywhere

git grep <string>

Clean unneeded files and optimize local repository

git gc

Create zip or tar file of a repository

Git archive –format=tar master

Delete objects without incoming pointers??

git prune

Identify corrupted objects

git fsck

Merge conflicts

Today I got this error during my usual git pull:

error: Pulling is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm '
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.

I did not wish to waste too much time. I tried a few things (git status, etc), none of which worked. As it is a small repository without much at stake and I know which files I changed, I simply deleted the clone and re-cloned the repo, put back the newer versions of the changed files, did a usual git add and git commit and git push. All was good. Not too much time wasted in becoming a gitmaster, a fate I wish to avoid.

Ignore a file

Put the unqualified name of the file in .gitignore at same level as .git. It will not be added to the project. Use this for keeping passwords secret.

References and related

https://www.freecodecamp.org/news/10-important-git-commands-that-every-developer-should-know/