Skip to content

feat: add optional pipeline-created snapshot processing#1259

Draft
arewm wants to merge 2 commits intokonflux-ci:mainfrom
arewm:use-build-snapshots
Draft

feat: add optional pipeline-created snapshot processing#1259
arewm wants to merge 2 commits intokonflux-ci:mainfrom
arewm:use-build-snapshots

Conversation

@arewm
Copy link
Copy Markdown
Member

@arewm arewm commented Aug 21, 2025

Implement support for build pipelines to create Snapshots directly rather than having Integration Service create them. This enables handling of complex use cases like bundle images, multiple artifacts, and single component mode abstraction while maintaining full backward compatibility.

Key changes:

  • Add GetSnapshotName() function to parse SNAPSHOT results from PipelineRuns
  • Enhance buildpipeline adapter to process existing snapshots when SNAPSHOT result present
  • Add pending annotation handling to skip ITS processing until snapshot is ready
  • Include comprehensive test coverage for both traditional and new workflows
  • Maintain 100% backward compatibility with existing snapshot creation workflow

The feature is opt-in based on presence of SNAPSHOT result in PipelineRun. When absent, the traditional snapshot creation workflow continues unchanged.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com
Signed-off-by: arewm arewm@users.noreply.github.com

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

Maintainers will complete the following section

@arewm arewm force-pushed the use-build-snapshots branch from d0a1c2f to ea47199 Compare August 21, 2025 19:35
Implement support for build pipelines to create Snapshots directly
rather than having Integration Service create them. This enables
handling of complex use cases like bundle images, multiple artifacts,
and single component mode abstraction while maintaining full backward
compatibility.

Key changes:
- Add GetSnapshotName() function to parse SNAPSHOT results from
  PipelineRuns
- Enhance buildpipeline adapter to process existing snapshots when
  SNAPSHOT result present
- Add pending annotation handling to skip ITS processing until snapshot
  is ready
- Include comprehensive test coverage for both traditional and new
  workflows
- Maintain 100% backward compatibility with existing snapshot creation
  workflow

The feature is opt-in based on presence of the SNAPSHOT result in the
PipelineRun. When absent, the traditional snapshot creation workflow
continues unchanged.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: arewm <arewm@users.noreply.github.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@arewm arewm force-pushed the use-build-snapshots branch from ea47199 to 9b78921 Compare August 21, 2025 19:47
Based on offline review feedback

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: arewm <arewm@users.noreply.github.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@konflux-ci-qe-bot
Copy link
Copy Markdown

@arewm: The following test has Failed, say /retest to rerun failed tests.

PipelineRun Name Status Rerun command Build Log Test Log
konflux-e2e-jszf6 Failed /retest View Pipeline Log View Test Logs

Inspecting Test Artifacts

To inspect your test artifacts, follow these steps:

  1. Install ORAS (see the ORAS installation guide).
  2. Download artifacts with the following commands:
mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/konflux-test-storage/konflux-team/integration-service:konflux-e2e-jszf6

Test results analysis

🚨 Failed to provision a cluster, see the log for more details:

Click to view logs
�[37mDEBU�[0m running 'mapt aws kind create'               
�[37mDEBU�[0m context initialized for mapt97d04a9d         
�[37mDEBU�[0m checking stack spotOption-kind-konflux-e2e-jszf6 
�[37mDEBU�[0m managing stack spotOption-kind-konflux-e2e-jszf6 
�[36mINFO�[0m Updating (spotOption-kind-konflux-e2e-jszf6): 
�[36mINFO�[0m                                              
�[36mINFO�[0m  +  pulumi:pulumi:Stack kind-konflux-e2e-jszf6-spotOption-kind-konflux-e2e-jszf6 creating (0s)  
�[37mDEBU�[0m Based on prices for instance types [c5a.8xlarge c6a.8xlarge c7i-flex.8xlarge c7i.8xlarge c6i.8xlarge c5ad.8xlarge c7a.8xlarge c6id.8xlarge] is az us-east-2b, current price is 0.63 with a score of 9 
�[37mDEBU�[0m Spot data: {[c5a.8xlarge c6a.8xlarge c7i-flex.8xlarge c7i.8xlarge c6i.8xlarge c5ad.8xlarge c7a.8xlarge c6id.8xlarge] 0.75828 us-east-2 us-east-2b 0} 
�[36mINFO�[0m @ updating......                             
�[36mINFO�[0m  +  rh:qe:aws:bso bso-bso creating (0s)      
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  rh:qe:aws:bso bso-bso created            
�[36mINFO�[0m  +  pulumi:pulumi:Stack kind-konflux-e2e-jszf6-spotOption-kind-konflux-e2e-jszf6 created (4s)  
�[36mINFO�[0m Outputs:                                     
�[36mINFO�[0m     az           : "us-east-2b"              
�[36mINFO�[0m     instancetypes: [                         
�[36mINFO�[0m         [0]: "c5a.8xlarge"                   
�[36mINFO�[0m         [1]: "c6a.8xlarge"                   
�[36mINFO�[0m         [2]: "c7i-flex.8xlarge"              
�[36mINFO�[0m         [3]: "c7i.8xlarge"                   
�[36mINFO�[0m         [4]: "c6i.8xlarge"                   
�[36mINFO�[0m         [5]: "c5ad.8xlarge"                  
�[36mINFO�[0m         [6]: "c7a.8xlarge"                   
�[36mINFO�[0m         [7]: "c6id.8xlarge"                  
�[36mINFO�[0m     ]                                        
�[36mINFO�[0m     max          : 0.75828                   
�[36mINFO�[0m     region       : "us-east-2"               
�[36mINFO�[0m     score        : 0                         
�[36mINFO�[0m                                              
�[36mINFO�[0m Resources:                                   
�[36mINFO�[0m     + 2 created                              
�[36mINFO�[0m                                              
�[36mINFO�[0m Duration: 5s                                 
�[36mINFO�[0m                                              
�[37mDEBU�[0m managing stack stackKind-kind-konflux-e2e-jszf6 
�[36mINFO�[0m Updating (stackKind-kind-konflux-e2e-jszf6): 
�[36mINFO�[0m                                              
�[36mINFO�[0m  +  pulumi:pulumi:Stack kind-konflux-e2e-jszf6-stackKind-kind-konflux-e2e-jszf6 creating (0s)  
�[36mINFO�[0m @ updating...........                        
�[36mINFO�[0m  +  aws:ec2:Vpc vpc-main-akd-net creating (0s)  
�[36mINFO�[0m  +  aws:ec2:Eip main-akd-lbeip creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:Eip main-akd-lbeip created (1s)  
�[36mINFO�[0m  +  tls:index:PrivateKey main-akd-pk creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:Vpc vpc-main-akd-net created (1s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-8888 creating (0s)  
�[36mINFO�[0m  +  aws:ec2:Subnet subnet-publicmain-akd-net0 creating (0s)  
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-6443 creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-8888 created (1s)  
�[36mINFO�[0m  +  aws:ec2:Subnet subnet-publicmain-akd-net0 created (1s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-22 creating (0s)  
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-6443 created (2s)  
�[36mINFO�[0m  +  tls:index:PrivateKey main-akd-pk created (3s)  
�[36mINFO�[0m  +  aws:ec2:SecurityGroup default-main-akd-net-main-akd-net creating (0s)  
�[37mDEBU�[0m Requesting a spot instance of types: c5a.8xlarge, c6a.8xlarge, c7i-flex.8xlarge, c7i.8xlarge, c6i.8xlarge, c5ad.8xlarge, c7a.8xlarge, c6id.8xlarge at us-east-2b paying: 0.758280 
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-22 created (1s)  
�[36mINFO�[0m  +  aws:ec2:InternetGateway igw-main-akd-net creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-9443 creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:SecurityGroup main-akd-sg creating (0s)  
�[36mINFO�[0m  +  aws:ec2:InternetGateway igw-main-akd-net created (2s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-9443 created (2s)  
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-8180 creating (0s)  
�[36mINFO�[0m  +  aws:ec2:SecurityGroup default-main-akd-net-main-akd-net created (4s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:LoadBalancer main-akd-lb creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:KeyPair main-akd-pk creating (0s)  
�[36mINFO�[0m  +  aws:lb:TargetGroup main-akd-tg-8180 created (1s)  
�[36mINFO�[0m  +  aws:ec2:SecurityGroup main-akd-sg created (3s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:KeyPair main-akd-pk created (1s)  
�[36mINFO�[0m  +  aws:ec2:RouteTable routeTable-publicmain-akd-net0 creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:LaunchTemplate main-akd-lt creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:RouteTable routeTable-publicmain-akd-net0 created (2s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:RouteTableAssociation routeTableAssociation-publicmain-akd-net0 creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:ec2:RouteTableAssociation routeTableAssociation-publicmain-akd-net0 created (0.88s)  
�[36mINFO�[0m @ updating.......                            
�[36mINFO�[0m  +  aws:ec2:LaunchTemplate main-akd-lt created (6s)  
�[36mINFO�[0m  +  aws:autoscaling:Group main-akd-asg creating (0s)  
�[36mINFO�[0m @ updating....................               
�[36mINFO�[0m  +  aws:autoscaling:Group main-akd-asg created (16s)  
�[36mINFO�[0m @ updating...................................................................................................................... 
�[36mINFO�[0m  +  aws:lb:LoadBalancer main-akd-lb created (142s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-22 creating (0s)  
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-6443 creating (0s)  
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-8888 creating (0s)  
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-9443 creating (0s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-22 created (0.81s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-6443 created (1s)  
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-8888 created (1s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-9443 created (2s)  
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-8180 creating (0s)  
�[36mINFO�[0m @ updating......                             
�[36mINFO�[0m  +  aws:lb:Listener main-akd-listener-8180 created (2s)  
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (0s)  
�[36mINFO�[0m @ updating..................                 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (15s) Dial 1/inf failed: retrying 
�[36mINFO�[0m @ updating..................                 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (30s) Dial 2/inf failed: retrying 
�[36mINFO�[0m @ updating...................                
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (45s) Dial 3/inf failed: retrying 
�[36mINFO�[0m @ updating..................                 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (61s) Dial 4/inf failed: retrying 
�[36mINFO�[0m @ updating..................                 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (76s) Dial 5/inf failed: retrying 
�[36mINFO�[0m @ updating........                           
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s)  
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) status: done 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) extended_status: done 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) boot_status_code: enabled-by-generator 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) last_update: Thu, 01 Jan 1970 00:00:55 +0000 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) detail: DataSourceEc2Local 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) errors: [] 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd creating (80s) recoverable_errors: {} 
�[36mINFO�[0m  +  command:remote:Command main-kind-readiness-akd-cmd created (80s) recoverable_errors: {} 
�[36mINFO�[0m @ updating....                               
�[36mINFO�[0m  +  command:remote:Command main... the content is too long - please download the artifact to see the full content

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants