| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-25 | |||
| 19:39:01 | mriedem | bauzas: dansmith: i take it from the comments here https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L150 that one shouldn't rely on ComputeNode.service_id is that correct? | |
| 19:39:13 | mriedem | the Service object has a compute_node field, and ComputeNode has a nullable service_id | |
| 19:39:23 | mriedem | but it looks like neither should really be used... | |
| 19:39:35 | dansmith | um, I dunno | |
| 19:40:35 | mriedem | ala https://github.com/openstack/nova/commit/551be2c52a29cb2755de4825a3fcb2c8f7d7b3f1 and https://github.com/openstack/nova/commit/0b177f18264ca842e16c2c989a081e773b3d86b7 | |
| 19:40:44 | mriedem | ok, lots of really old compat notes from bauzas in all of that code | |
| 19:41:02 | mriedem | going back to kilo/liberty era | |
| 19:41:09 | mriedem | wonder if we'll ever clean that stuff up | |
| 19:41:11 | dansmith | yeah, | |
| 19:41:27 | dansmith | I'd think that that comment is probably old and that a migration fixed the ambiguity long ago, but.. I haven't looked | |
| 19:43:51 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L123 | |
| 19:43:58 | mriedem | i don't think there was ever a migration | |
| 19:44:02 | mriedem | just lots of todos from ~liberty | |
| 19:45:57 | dansmith | hrm | |
| 19:45:59 | mriedem | ok i'd think the migration would be pretty simple, | |
| 19:46:08 | mriedem | iterate cells looking for compute_nodes where host is null | |
| 19:46:18 | mriedem | if so, find their service record and update the compute_nodes.host value | |
| 19:46:55 | mriedem | or a migration run per-cell (we haven't really be consistent there) | |
| 19:46:59 | dansmith | or just have computes update compute nodes on RT update | |
| 19:47:10 | dansmith | no reason to make it a manual-only thing | |
| 19:48:00 | mriedem | yeah i don't think it's even possible to get through RT with a cn that doesn't have a host value set | |
| 19:48:14 | mriedem | we lookup the cn by host and node and if it doesn't exist we create it and always have those values set now | |
| 19:48:34 | mriedem | so...maybe this is as good as a simple nova-status upgrade check to sniff out anything lingering and in U drop all of this compat code | |
| 19:48:43 | dansmith | sure | |
| 19:48:48 | mriedem | not sure we'd want to do the db alter table statements to make the column non-nullable... | |
| 19:48:54 | mriedem | but that's less of an issue | |
| 20:03:06 | efried | mriedem: upt also gets called on spawn & delete, doesn't it? | |
| 20:03:23 | efried | via update_available_resource | |
| 20:05:36 | mriedem | efried: i don't think so, you're thinking of instance_claim | |
| 20:05:58 | efried | okay | |
| 20:06:19 | mriedem | and rt.update_usage | |
| 20:06:57 | mriedem | ^ is the one that has the vm_state check for DELETED/SHELVED_OFFLOADED on the instance and if it's one of those it removes the usage | |
| 20:07:34 | mriedem | looks like _post_live_migration also hits update_available_resource for kicks | |
| 20:08:04 | mriedem | presumably b/c we don't do rt claims for live migration | |
| 20:12:25 | efried | okay, so chattier than just once every 10 minutes. | |
| 20:14:02 | sean-k-mooney | mriedem: well we do in some cases | |
| 20:14:51 | sean-k-mooney | mriedem: we claim pci devices for the sriov case and we will claim cpus and hugepages once artoms numa migration code lands | |
| 20:15:10 | mriedem | update_available_resource is once per minute i think | |
| 20:15:31 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval | |
| 20:15:55 | mriedem | https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py#L31 | |
| 20:16:04 | sean-k-mooney | its configurable in the config but i thought it was 300 seconds | |
| 20:16:38 | mriedem | i think you're both high on goofballs | |
| 20:17:00 | mriedem | efried: so every minute by default | |
| 20:17:21 | sean-k-mooney | on no its 60s you are right https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval | |
| 20:18:26 | sean-k-mooney | oh wait is that only set for the schduler ? i though we had a compute node version too? | |
| 20:24:42 | mriedem | what are you asking | |
| 20:24:59 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.update_resources_interval is used by the ComputeManager.update_available_resource periodic task | |
| 20:40:13 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Re-propose the spec to allow specifying a list of CPU models https://review.opendev.org/642030 | |
| 21:38:44 | openstackgerrit | sean mooney proposed openstack/nova-specs master: add libvirt pqos spec https://review.opendev.org/662264 | |
| 21:39:24 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Add spec for expose-auto-converge-post-copy https://review.opendev.org/651681 | |
| 21:52:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add placement request filter for disabled computes https://review.opendev.org/654596 | |
| #openstack-nova - 2019-06-26 | |||
| 01:31:18 | openstackgerrit | Merged openstack/nova master: hacking: Resolve W503 (line break occurred before a binary operator) https://review.opendev.org/651555 | |
| 01:31:27 | openstackgerrit | Merged openstack/nova master: hacking: Resolve E741 (ambiguous variable name) https://review.opendev.org/652103 | |
| 01:54:00 | openstackgerrit | Yongli He proposed openstack/nova-specs master: grammar fix for show-server-numa-topology spec https://review.opendev.org/667487 | |
| 01:57:14 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.opendev.org/648912 | |
| 01:57:15 | openstackgerrit | Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765 | |
| 02:27:28 | openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/612626 | |
| 02:44:36 | openstackgerrit | Alex Xu proposed openstack/nova master: Correct the comment of RequestSpec's network_metadata https://review.opendev.org/667061 | |
| 05:22:19 | openstackgerrit | Yongli He proposed openstack/nova-specs master: grammar fix for show-server-numa-topology spec https://review.opendev.org/667487 | |
| 06:43:31 | openstackgerrit | Merged openstack/nova master: Remove comments about mirroring changes to nova/cells/messaging.py https://review.opendev.org/667107 | |
| 06:43:42 | openstackgerrit | Merged openstack/nova master: Drop source node allocations if finish_resize fails https://review.opendev.org/654067 | |
| 06:52:59 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.74: Support Specifying AZ to unshelve https://review.opendev.org/665136 | |
| 07:29:24 | brinzhang | efried: Are you around? | |
| 08:25:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: pull out functions from _heal_allocations_for_instance https://review.opendev.org/655457 | |
| 08:25:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: reorder conditions in _heal_allocations_for_instance https://review.opendev.org/655458 | |
| 08:25:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prepare _heal_allocations_for_instance for nested allocations https://review.opendev.org/637954 | |
| 08:25:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: pull out put_allocation call from _heal_* https://review.opendev.org/655459 | |
| 08:25:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955 | |
| 09:04:09 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Grab fresh power state info from the driver https://review.opendev.org/665975 | |
| 09:29:12 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Update AZ admin doc to mention the new way to specify hosts https://review.opendev.org/666767 | |
| 09:32:45 | kashyap | Does anyone here of an existing bug in the Gate, where the "tempest-slow-py3" is failing with: | |
| 09:32:48 | kashyap | tempest.exceptions.BuildErrorException: Server 008c5c50-ff54-49f4-adb0-23775e8af5f1 failed to build and is in ERROR status | |
| 09:32:51 | kashyap | Details: {'code': 500, 'created': '2019-06-25T20:55:49Z', 'message': 'Unexpected vif_type=binding_failed'} | |
| 09:32:54 | kashyap | http://logs.openstack.org/89/667389/1/check/tempest-slow-py3/2606bcc/testr_results.html.gz | |
| 09:32:59 | kashyap | [That is a stable/stein backport] | |
| 09:37:21 | kashyap | Okay, I see time outs (also in the stable/rocky backport) in the 'testr_results'. /me goes to 'recheck' | |
| 11:06:46 | NewBruce | Hey @kayshap | |
| 11:07:47 | NewBruce | so good news, i didnt try to mess around with xml, instead just use SELinux ;) but not sure if you can help out on this one - migrations are failing with | |
| 11:07:48 | NewBruce | Live Migration failure: Library function returned error but did not set virError: libvirtError: Library function returned error but did not set vir | |
| 11:08:17 | NewBruce | digging into the libvirt logs - | |
| 11:08:17 | NewBruce | 2019-06-26 09:46:22.816+0000: 30621: error : virNetClientStreamRaiseError:200 : stream had I/O failure | |
| 11:08:17 | NewBruce | 2019-06-26 09:46:23.190+0000: 19228: error : virNetClientProgramDispatchError:177 : internal error: qemu unexpectedly closed the monitor: 2019-06-26T09:46:22.815029Z qemu-kvm: Failed to load PCIDevice:config | |
| 11:08:17 | NewBruce | 2019-06-26T09:46:22.815033Z qemu-kvm: Failed to load virtio-net:virtio | |
| 11:08:17 | NewBruce | 2019-06-26T09:46:22.815036Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net' | |
| 11:09:34 | NewBruce | thing is, from the control side of things, the migration completed - no errors or anything are returned… also, i could migrate fine between these hosts before i added SELinux, and it (rarely) works to migrate a machine… im lost if its a LibVirt or nova issue at this point - thoughts? | |
| 11:10:12 | NewBruce | the osc reports life is peachy : openstack server migrate --live cc-compute28-sto2 aadfe56a-88b8-49c0-9dac-41a4c494c1b5 --wait | |
| 11:10:12 | NewBruce | Progress: 97Complete | |
| 11:11:25 | NewBruce | but nova never gets to post-migration, and i dont think its actually doing the migration itself - on the source | |
| 11:11:25 | NewBruce | Took 2.35 seconds for pre_live_migration on destination host cc-compute26-sto2. | |
| 11:11:25 | NewBruce | Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) | |
| 11:11:25 | NewBruce | Migration operation has aborted | |
| 12:27:17 | mriedem | lyarwood: is your plan for https://bugs.launchpad.net/nova/+bug/1832248 to get https://review.opendev.org/#/c/664418/ released and then bump the lower constraint dependency from nova to os-brick and then consider the nova bug fixed? | |
| 12:27:18 | openstack | Launchpad bug 1832248 in OpenStack Compute (nova) "tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing when using the Q35 machine type" [Undecided,New] | |
| 12:32:56 | alex_xu | mriedem: hope we answered all your question https://review.opendev.org/601596, looking for one more +2 :) | |
| 12:33:38 | alex_xu | johnthetubaguy: ^ hope you around, the vpmem spec is in good shape | |
| 12:34:19 | lyarwood | mriedem: no the nova bug is seperate, the os-brick change just works around the underlying QEMU issue Nova is hitting with q35 | |
| 12:36:31 | mriedem | lyarwood: oh ok | |
| 12:36:49 | mriedem | alex_xu: ack, i still need to read all of the replies... | |
| 12:37:20 | alex_xu | mriedem: hah, i see, a lot | |
| 12:39:45 | alex_xu | mriedem: also, there is the code for reference https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory, althought it is merge conflict, but it is still good to see what we probably are going to change in the code | |