| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-02 | |||
| 17:42:57 | sean-k-mooney | not in the libvirt dirver | |
| 17:43:10 | sean-k-mooney | you can do that with qemu but not via nova | |
| 17:43:13 | Sundar | libvirt allows it -- but apparently not the Nova libvirt driver? | |
| 17:43:27 | Sundar | I see | |
| 17:43:38 | sean-k-mooney | yes becasue we have no where to store the ram dump | |
| 17:44:31 | sean-k-mooney | we could in theory upload it as a second image to glace or something but snapshot in the nova api is just of the disk | |
| 17:46:00 | Sundar | Are you able to check snapshot with FPGA-attached VMs? I'll try too -- but what works in the lab may not always work in the field. | |
| 17:46:41 | sean-k-mooney | i have not deployed on real hardware so no. | |
| 17:48:40 | Sundar | I would say that it is safer to block something rather than speculatively allow it. Do you have users who strongly need snapshots with accelerators? | |
| 17:49:43 | sean-k-mooney | not specifically but i would not consider movingit form techpreview without it | |
| 17:50:10 | sean-k-mooney | in the curent patch there are no ways to do any move operation and you cant resuceu or snapshot the vm | |
| 17:50:28 | sean-k-mooney | so there is 0 way to do maintaince on the host or recover in the case of a vailure | |
| 17:50:33 | sean-k-mooney | even evacuate is blocked | |
| 17:50:58 | sean-k-mooney | as is adding or removeing volumes or interfaces | |
| 17:51:22 | sean-k-mooney | so with only the abilty to boot and delete a vm we would not support this in productin for customer to use | |
| 17:52:21 | sean-k-mooney | thats a downstrema desciion however. it wont affect if we support this in upstream | |
| 17:52:56 | Sundar | I see your concern. The real solution is to test these ops well with real accelerators and real-ish workloads (not sample bitstreams). I'll pass that on to my Cyborg/Intel colleagues. | |
| 17:53:22 | Sundar | Meanwhile, based on your input, I could remove the blocks on snapshot and rescue. Presumably backup too? | |
| 17:54:16 | sean-k-mooney | backup is basicaly a cron job in the nova code that calls snapshot and clean up old snapshots | |
| 17:54:27 | sean-k-mooney | so if snapshot work then back will | |
| 17:55:24 | sean-k-mooney | im not sure about rescue. it should work but i dont know if the acclerator will be aviable while its in resuce state | |
| 17:55:53 | sean-k-mooney | if the accelerator in not present in the vm during or after resuce has been completed then we might need to block it | |
| 17:56:32 | Sundar | Since the rescue brings in a different disk image, which may not have the right device drivers, it probably will not. The unrescue to the previous image "should" work. | |
| 17:57:04 | sean-k-mooney | oh the other hand not everyone would need the acclerator to be present during resuce so as long as we dont loose itafter unrescue we should be ok | |
| 17:57:23 | Sundar | Ok. So rescue is in. What about suspend? It doesn't work today because we don't have the code to handle accelerators in that path. | |
| 17:57:51 | sean-k-mooney | suspend wont work without a code change so ya that can remain blocked | |
| 17:58:31 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (claims) https://review.opendev.org/710847 | |
| 17:58:32 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (refresh) https://review.opendev.org/710848 | |
| 17:59:23 | Sundar | sean-k-mooney: The other 2 ops mentioned in https://review.opendev.org/#/c/674726/3/nova/tests/functional/test_servers.py are rebuild and resize. | |
| 17:59:42 | dansmith | Sundar: sean-k-mooney just catching up, but if we can't support snapshot, that's a huge problem IMHO (although I like sean-k-mooney think it's likely not a problem) | |
| 18:00:19 | dansmith | if rescue doesn't include the accelerator I don't think we should block it | |
| 18:00:33 | dansmith | rescue is a special environment | |
| 18:00:46 | dansmith | I don't really know why it wouldn't include it, but I don't think that's a major problem or anything | |
| 18:01:02 | sean-k-mooney | ya i think rescue should be fine just as long as we validate the acclertor is present after unrescue when we go back to the stanard env | |
| 18:01:21 | dansmith | yes, that's obviously an important point :D | |
| 18:01:34 | sean-k-mooney | Sundar: resize wont work | |
| 18:01:49 | sean-k-mooney | i tested that with the fake dirver and it does nto update placmenet correctly | |
| 18:01:57 | sean-k-mooney | rebuild should work fine | |
| 18:04:01 | sean-k-mooney | Sundar: dansmith i left comment on https://review.opendev.org/#/c/674726 with what i think should and should not work | |
| 18:10:23 | sean-k-mooney | Sundar: by the way im going to be AFK or on PTO for most of the rest of this week so if i dont repond or am offline that is why | |
| 18:17:38 | Sundar | sean-k-mooney: dansmith: I'll respond in the patch. Meanwhile I have filed a Cyborg storyboard task: https://storyboard.openstack.org/#!/story/2007358 . I'll raise that with Cyborg folks. | |
| 18:18:34 | sean-k-mooney | Sundar: is there a thirdpart hardwar ci somewhere? | |
| 18:18:54 | sean-k-mooney | if so they could just do what im doing in the multinode job | |
| 18:19:15 | sean-k-mooney | that will validate all the lifecycle ops | |
| 18:19:55 | sean-k-mooney | they could blacklist the tests they know do not work and as the missing ops get adressed they can be removed form the blacklist | |
| 18:20:19 | Sundar | sean-k-mooney: Unfortunately, our current plans for 3pCI with FPGAs are nixed. Our colleagues in other parts of Intel could possibly resume it, since we have the hardware lined up. | |
| 18:20:24 | sean-k-mooney | alternitvly the cyborg plugin could be extended but that really up to whowever is going to test it | |
| 18:20:47 | sean-k-mooney | ok ya i kind of guessed that would be the case | |
| 18:22:25 | Sundar | sean-k-mooney: Noted your PTO. Thanks for your help. I'm not sure how long I'll be around after this week. | |
| 18:34:29 | sean-k-mooney | ok time for dinner. | |
| 18:34:43 | sean-k-mooney | o/ | |
| 19:33:25 | openstackgerrit | Merged openstack/nova master: Fix os-attach-interfaces policy to be admin_or_owner https://review.opendev.org/705135 | |
| 19:50:53 | openstackgerrit | David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861 | |
| 19:51:09 | openstackgerrit | David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861 | |
| 20:21:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Use virDomainBlockCopy to swap volumes with >= 5.10.0 https://review.opendev.org/696834 | |
| 20:21:52 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701 | |
| 20:30:21 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-aggregates policies https://review.opendev.org/701651 | |
| 20:48:26 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-assisted_volume_snapshots policies https://review.opendev.org/701835 | |
| 21:37:56 | redrobot | Hi Nova friends! I'd love to get another set of eyes on https://review.opendev.org/#/c/680562/ | |
| 21:58:53 | mnaser | what's the best way of forcing an instance type to specifically boot on a specific host now? if i use the filter which uses an aggregate's instance_type, i would have to put all the others in an aggregate with instance_type including all the other instance types | |
| 21:59:52 | mnaser | the only issue is that in this case, i have no simple way of enforcing no other instance type than $foo lands on this vm | |
| 22:01:03 | mnaser | if i use traits, i have to use forbidden/required but that means if someone creates a flavor and forgets to add the forbidden, it'll end up on that host possibly | |
| 22:08:32 | dansmith | mnaser: does this help? | |
| 22:08:33 | dansmith | https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html | |
| 22:08:39 | dansmith | that was the point of that work, IIRC | |
| 22:10:37 | mnaser | dansmith: ah that's perfect, sorry i tried looking around but did not find that | |
| 22:10:47 | dansmith | it's recent | |
| 22:13:56 | mnaser | neat, thank you. | |
| #openstack-nova - 2020-03-03 | |||
| 00:00:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Fix os-volumes-attachments policy to be admin_or_owner https://review.opendev.org/709955 | |
| 00:03:38 | joshualyle | does anyone have any ideas on why my compute nodes would only seem to schedule migrations between subsets of nodes? | |
| 00:04:10 | joshualyle | I upgraded half of my nodes and then the other half but now the latter half will only live-migrate between themselves and I'm not sure why | |
| 00:08:04 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929 | |
| 00:08:21 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388 | |
| 00:16:59 | openstackgerrit | Merged openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682 | |
| 00:19:53 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929 | |
| 00:21:03 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388 | |
| 00:23:14 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-assisted_volume_snapshots policy https://review.opendev.org/701837 | |
| 00:23:21 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-assisted_volume_snapshots policies https://review.opendev.org/701840 | |
| 00:23:32 | openstackgerrit | Merged openstack/nova master: api-ref: mark device response param as optional for list/show vol attachments https://review.opendev.org/690383 | |
| 00:25:57 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 00:46:38 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 01:10:38 | joshualyle | turns out it might just be overallocated because of some weirdness during previous allocations. Is there a straightforward way to fix "There are allocations remaining against the source host that might need to be removed" | |
| 01:16:06 | melwitt | joshualyle: this might be of help https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html | |
| 01:40:04 | joshualyle | thank you melwitt. I was able to clean up some messed up allocations but I still have a host that refuses to accept any migrations :-/ | |
| 01:55:54 | melwitt | joshualyle: I would get the osc-placement cli (it's a plugin for osc) and use this command to see if there's any allocations that don't belong on it https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show | |
| 01:56:16 | melwitt | openstack resource provider show --allocations <uuid> | |
| 01:57:15 | melwitt | you can also use that cli to remove bad allocations. just be aware that it does a full replace when you 'set' allocations. so removing allocations is by omission of only the ones you want gone and setting only the ones you want to keep | |
| 02:13:35 | joshualyle | melwitt: I'm currently on rocky. "openstack resource provider" doesn't seem to be available | |
| 02:14:29 | joshualyle | after a lot of troubleshooting, the problem seems to be clock drift because chrony wasn't working and neutron not working correctly on the destination compute node causing preflight check failures | |
| 02:18:51 | openstackgerrit | Merged openstack/nova master: Workaround a race initialising version control in db_version() https://review.opendev.org/619622 | |
| 02:19:01 | openstackgerrit | Merged openstack/nova master: Add description of live_migration_timeout_action option https://review.opendev.org/700124 | |
| 02:19:11 | openstackgerrit | Merged openstack/nova master: doc: note the need to configure cinder auth in reclaim_instance_interval https://review.opendev.org/690602 | |
| 02:19:20 | openstackgerrit | Merged openstack/nova master: Monkey patch original current_thread _active https://review.opendev.org/707474 | |
| 04:21:10 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242 | |
| 04:21:11 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | |
| 04:21:11 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | |
| 04:21:12 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581 | |
| 04:21:12 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245 | |
| 04:21:13 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | |
| 04:21:13 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable hard/soft reboot with accelerators. https://review.opendev.org/697940 | |