| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 19:03:24 | sean-k-mooney | so "import nova.conf; from nova import config; CONF = nova.conf.CONF; config.parse_args(sys.argv)" | |
| 19:03:41 | sean-k-mooney | cz3: if you do that i think it should work | |
| 19:04:59 | sean-k-mooney | although im not sure you need to initalise rpc or configre the db | |
| 19:05:21 | cz3 | in this case I just need the db access | |
| 19:06:49 | sean-k-mooney | which looks like its jsut sqlalchemy_api.configure(CONF) | |
| 19:07:26 | cz3 | sean-k-mooney: config.parse_args did the trick | |
| 19:07:31 | cz3 | thank you! | |
| 19:07:31 | sean-k-mooney | in any case i would use https://github.com/openstack/nova/blob/master/nova/cmd/manage.py as a reference | |
| 19:07:43 | sean-k-mooney | no worries | |
| 19:08:03 | sean-k-mooney | CONF(argv[1:], | |
| 19:08:05 | sean-k-mooney | project='nova', | |
| 19:08:07 | sean-k-mooney | version=version.version_string(), | |
| 19:08:09 | sean-k-mooney | default_config_files=default_config_files) | |
| 19:08:11 | sean-k-mooney | that is likely the bit you really needed | |
| 19:08:37 | sean-k-mooney | that is invokeing the gloabl constuctor for the singelton | |
| 19:09:11 | cz3 | I actually aim to add something to nova-manage and send it over for review, just wanted to test few things beforehand | |
| 19:09:35 | cz3 | I emailed the openstack-discuss list last week about this, it was re: request_specs regeneration | |
| 19:09:56 | cz3 | where I wanted to find a way to sync extra_specs from flavor data into existing instances | |
| 19:10:33 | sean-k-mooney | oh yes i remember | |
| 19:10:39 | cz3 | long time ago I did something like this by directly appending stuff into request_spec json in database, but this time I want to do it the proper way | |
| 19:13:20 | cz3 | I know that flavors are supposed to be immutable etc. but in this case I just want the existing request_specs to reflect the actual reality and not make the scheduler migrate instances where it was not supposed to after underlying flavor was updated with extra_specs | |
| 19:15:05 | cz3 | I will try to add that to nova-manage and send it over to review via gerrit, maybe it will be good enough to get upstreamed | |
| 19:15:54 | sean-k-mooney | the main consurn we woudl have it that such an update can lead to the vm being broken or on an invalid host also | |
| 19:16:30 | sean-k-mooney | on one hand nova manage is admin only and require direct acess to the server hosting the cloud to get the configs | |
| 19:16:59 | sean-k-mooney | but from a downstream product point of view im not sure we would support customer executing this command | |
| 19:17:16 | sean-k-mooney | we may be with the caveat tha that if you break things you fix them | |
| 19:17:33 | sean-k-mooney | its something we would have to think about | |
| 19:18:01 | cz3 | yeah, I do share the concern, I don't want to send something that would end up breaking deployments for other people if used improperly | |
| 19:18:35 | sean-k-mooney | you proably would need to run som of the flavor/image compatiabity checks before doing the update to make it safe | |
| 19:18:53 | sean-k-mooney | our add a --yes-i-really-really-mean-it flag | |
| 19:19:14 | sean-k-mooney | anyway o/ time to go have dinner | |
| 19:19:42 | cz3 | sure | |
| 19:20:05 | cz3 | thanks again! | |
| 21:57:38 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: zuul: Start to migrate nova-live-migration to zuulv3 https://review.opendev.org/c/openstack/nova/+/795432 | |
| 21:57:39 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs https://review.opendev.org/c/openstack/nova/+/795374 | |
| 22:07:55 | tosky | melwitt: an supersquash! Would it make sense to change the first line of the commit message? It is more than just that single commit it was taken from | |
| 22:21:29 | opendevreview | melanie witt proposed openstack/nova master: api: Log correct client IP if load balancer in use https://review.opendev.org/c/openstack/nova/+/786766 | |
| 22:25:02 | gmann | stephenfin: can you check if anything else you want to add in this spec, leaving +w to you https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| #openstack-nova - 2021-06-10 | |||
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012 | |
| 00:02:37 | opendevmeet | bug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/695187 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/695188 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 00:02:38 | opendevreview | melanie witt proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 06:39:39 | masterpe[m] | I have more resources declined in the placement.allocations table then actual in use. I see some instance ID there that are deleted. | |
| 06:39:56 | masterpe[m] | What shall I do with them? | |
| 06:46:16 | frickler | masterpe[m]: seems this tool was made for you https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-audit | |
| 06:46:53 | frickler | or maybe the heal-allocations above | |
| 06:52:57 | masterpe[m] | nice thanks | |
| 08:06:14 | lyarwood | kashyap: https://review.opendev.org/c/openstack/nova/+/795533/5#message-3e7b1dc4d3f8d22b2c9a55637c7366199d00eb56 - I'll write this up in a Nova bug and likely libvirt bug later today but would you mind scanning this if you get a chance? | |
| 08:06:37 | kashyap | lyarwood: Mornin; /me clicks | |
| 08:07:30 | kashyap | lyarwood: Ah, you're debugging the informative error, "reason=failed" | |
| 08:07:45 | lyarwood | there's two parts really | |
| 08:07:51 | lyarwood | yeah the reason=failed thing | |
| 08:08:18 | lyarwood | and the way that the call to virDomainMigrateToURI3 from the source doesn't pick up the failure on the dest | |
| 08:09:23 | kashyap | Hmm, yikes | |
| 08:09:25 | lyarwood | I'm assuming that the dest tears down the connection between the two leading to the eventual `unable to connect to server` error | |
| 08:10:09 | lyarwood | instead of the source being told the migration has failed etc | |
| 08:10:47 | kashyap | lyarwood: This is the dest, right: https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/controller/logs/libvirt/libvirtd_log.txt | |
| 08:10:54 | lyarwood | yeah | |
| 08:11:15 | kashyap | Is it just me or is it loading deadly slow? I just want to pull down the entire raw file | |
| 08:11:50 | lyarwood | yeah these files are pretty large, I typically pull them down now | |
| 08:12:02 | kashyap | My browser is hung here (probably it goes into MBs). Do I clicck on the "View log" to get the raw link? | |
| 08:12:15 | lyarwood | it's awkward as the raw links don't use .gz at the end of the file names | |
| 08:12:17 | kashyap | Maybe I can simply `wget` the above URL - /me tries | |
| 08:12:37 | lyarwood | mv it to a .gz file and either gunzip or let vim unpack them btw | |
| 08:12:40 | kashyap | Right, but it _is_ a .gz file - a new user will discover it after your browser crashes :D | |
| 08:12:53 | kashyap | lyarwood: Yeah; that's become "muscle memory" now | |
| 08:12:54 | lyarwood | browsers can handle .gz | |
| 08:14:08 | kashyap | Oh, sure; sometimes, a very large file just crashed FF for me | |
| 08:15:31 | kashyap | Actual link to `wget` is: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/controller/logs/libvirt/libvirtd_log.txt | |
| 08:15:36 | kashyap | 93MB only | |
| 08:32:27 | kashyap | lyarwood: Two quick things: is this reproducible? Or is it the first you noticed? | |
| 08:32:42 | lyarwood | It's the first time I've seen this | |
| 08:32:52 | kashyap | lyarwood: Also, strangely, I don't see the migrateToURI3() failure in the source libvirtd log: | |
| 08:32:57 | kashyap | I pulled the 113MB file from here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/compute1/logs/libvirt/libvirtd_log.txt | |
| 08:33:12 | kashyap | I wonder if it got log-rotated | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Update Servers APIs https://review.opendev.org/c/openstack/nova/+/764292 | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replace all_tenants with all_projects in List Server APIs https://review.opendev.org/c/openstack/nova/+/765311 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Flavor Access APIs https://review.opendev.org/c/openstack/nova/+/767704 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 08:34:46 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:38:27 | lyarwood | 93717 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobEventProcessConcluded:1489 : handling job 'drive-virtio-disk0' state '3' newstate '0' | |
| 08:38:27 | lyarwood | 93718 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobProcessEventConcludedCopyAbort:1250 : copy job 'drive-virtio-disk0' on VM 'instance-0000001d' aborted | |
| 08:38:35 | lyarwood | looks like the block job failed | |
| 08:39:36 | lyarwood | 93774 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:41:08 | kashyap | lyarwood: Yeah; that's a good find | |
| 08:41:47 | kashyap | The thing is - why the connection is refused? I want to think of "firewall", but I don't think that's it | |
| 08:42:02 | lyarwood | didn't we have issues with test_live_block_migration_paused before? | |
| 08:42:07 | lyarwood | kashyap: I don't think it's refused | |
| 08:42:30 | lyarwood | kashyap: the migration just ends and the python libvirt lib is just incorrectly handling the failure | |
| 08:42:32 | kashyap | lyarwood: Oh, right: | |
| 08:42:41 | kashyap | It's because the mirroring was cancelled on the source: | |
| 08:42:43 | kashyap | --- | |