| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-26 | |||
| 12:32:59 | sean-k-mooney | why do we need to use legacy requests specs in this case? | |
| 12:33:38 | sean-k-mooney | i taught we had already moved to ovo form in rocky but i guess not fully | |
| 12:36:32 | openstack | bug 1529084 in oslo.messaging "RPC fake driver should accept datetime items for data" [Undecided,Fix released] https://launchpad.net/bugs/1529084 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:36:32 | sean-k-mooney | oh its becasue of the workaround for bug #1529084 | |
| 12:37:19 | sean-k-mooney | that intoduced the seriasiation and deserialisation but missed the fact it wont recurse to the nested ovos which is what you are fixing | |
| 12:39:03 | sean-k-mooney | another fix would be to fix the rpc fake dirver to accept datimes in dicst and remvoe the jsonutils calls entirely | |
| 12:39:47 | gibi | sean-k-mooney: since stein we doesn't use the legacy dict but pass around the ovo, but that is an RPC change it is not backportable | |
| 12:40:33 | gibi | sean-k-mooney: but yeah, we we can somehow remove the need of the json.loads(json.dumps(..) stuff that would also help | |
| 12:41:09 | sean-k-mooney | well the comment says its done beacase teh fake rpc driver does not support datetimes in dicts | |
| 12:41:16 | sean-k-mooney | so if we fix that then we could remove it right | |
| 12:41:31 | sean-k-mooney | i assume there was a reason bauzas didnt do that orgininally | |
| 12:41:43 | sean-k-mooney | other then this was faster | |
| 12:42:35 | gibi | I'm not sure what will happen if the dump an load is removed and then the rpc will try to serialize the dict again with ovos inside | |
| 12:42:48 | sean-k-mooney | gibi: anway i have not checked that setting default=... will work but your change looks resonable to me | |
| 12:43:43 | gibi | sean-k-mooney: the solution is basically change how the to_primitive works during dumps to trigger the ovo serialization instead of stuck in an infinite loop | |
| 12:44:09 | sean-k-mooney | ya i was assuming "default=jsonutils.to_primitive" if not set | |
| 12:44:14 | openstack | Launchpad bug 1864678 in oslo.serialization "jsonutils.to_primitive does not follow the protocol required by json.dump" [Medium,Triaged] | |
| 12:44:14 | gibi | I also filed a bug to oslo.serialization https://bugs.launchpad.net/oslo.serialization/+bug/1864678 | |
| 12:44:24 | sean-k-mooney | so you are just passing an extra argment to it with functools.partial | |
| 12:44:32 | gibi | sean-k-mooney: yeas, that is the default by default (sic) | |
| 12:44:38 | gibi | sean-k-mooney: yes | |
| 12:45:18 | sean-k-mooney | yep so that all makes sense to me. its not the cleanest thing long term but its the minimal backportable change which is more important | |
| 12:46:03 | openstack | Launchpad bug 1529084 in oslo.messaging "RPC fake driver should accept datetime items for data" [Undecided,Fix released] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:46:03 | gibi | I'm not sure that the fix of https://bugs.launchpad.net/oslo.messaging/+bug/1529084 can be backported to pike | |
| 12:46:35 | gibi | and the the we can bump oslo.messaging version on the stable branches | |
| 12:47:03 | gibi | then remove the loads(dumps(..)) part from nova, and backport it to pike | |
| 12:47:05 | sean-k-mooney | you mean to rocky | |
| 12:47:17 | gibi | sean-k-mooney: rocky is the first stable that is affected | |
| 12:47:25 | gibi | I have to backport the whole thing to pike | |
| 12:47:28 | sean-k-mooney | ah you have other cherry picks | |
| 12:47:32 | gibi | as we have a failure downstream | |
| 12:47:40 | gibi | in pike | |
| 12:47:47 | sean-k-mooney | gotch ya | |
| 12:48:16 | gibi | so my target is to fix pike, and sure I will fix every broken stable along the way but I wan't to do this with minimal change | |
| 12:48:31 | gibi | a new oslo.messaging version would be a lot harder business | |
| 12:48:32 | sean-k-mooney | not a core but i +1'd it anyway since it makes sense after our disucsstion stephenfin its pretty quick if you can take a look | |
| 12:48:42 | gibi | sean-k-mooney: thanks! | |
| 12:53:25 | sean-k-mooney | hehe i some how dont think the intel pmem ci is going to work on rocky | |
| 12:53:40 | sean-k-mooney | they might want to fix that at some point | |
| 12:55:27 | openstack | bug 1864776 in OpenStack Compute (nova) "os-volumes-attachments API policy is allowed for everyone even policy defaults is admin_or_owner" [Undecided,In progress] https://launchpad.net/bugs/1864776 - Assigned to Brin Zhang (zhangbailin) | |
| 12:55:27 | brinzhang | gmann: is that true of bug 1864776? | |
| 13:01:43 | brinzhang | gmann: I was pushed the fixed patch https://review.opendev.org/#/c/709955/, and rebase this patch I was tested https://review.opendev.org/#/c/709929/1/nova/tests/unit/policies/test_volumes.py, it also need everyone contexts to authorize the admin_or_owner role | |
| 13:02:13 | brinzhang | gmann: I am not sure this bug is true, while you are free, pls review, thanks | |
| 13:48:21 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Don't print user_data for 'nova show' https://review.opendev.org/708850 | |
| 13:50:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add framework for extra spec validation https://review.opendev.org/704643 | |
| 13:50:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add documentation for flavor extra specs https://review.opendev.org/710037 | |
| 13:50:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add microversion 2.82, extra spec validation https://review.opendev.org/708436 | |
| 13:54:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Hey let's support routed networks y'all! https://review.opendev.org/656885 | |
| 13:55:55 | openstack | Launchpad bug 1864776 in OpenStack Compute (nova) "os-volumes-attachments API policy is allowed for everyone even policy defaults is admin_or_owner" [Undecided,In progress] - Assigned to Brin Zhang (zhangbailin) | |
| 13:55:55 | brinzhang__ | stephenfin: do you have time the check this bug? That prevent my next work https://bugs.launchpad.net/nova/+bug/1864776 | |
| 13:56:21 | brinzhang__ | that's the default policy refresh patch | |
| 13:57:50 | brinzhang__ | stephenfin: if it's true, I will rebase others on the fixed patch, otherwise I will abandon this fix, and push new patc to go continue | |
| 14:00:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Reintroduce volume based LM tests https://review.opendev.org/536105 | |
| 14:05:21 | canori01 | hey guys, is it possible to update the root disk for an in-use flavor in the database (such that new instances using that flavor take the change)or would that break things? | |
| 14:09:54 | brinzhang__ | canori01: Which things would you like to change of the root disk? | |
| 14:11:29 | brinzhang__ | canori01:It is not recommended to change the configuration of the root disk directly in db. | |
| 14:12:16 | brinzhang__ | May cause strange things due to incomplete modification. | |
| 14:21:26 | canori01 | brinzhang: Only thing I want to change is the root disk size from 0 to something like 20 or 50G | |
| 14:33:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix incorrect vm and task state after build failure race https://review.opendev.org/689388 | |
| 14:33:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382 | |
| 14:50:42 | openstackgerrit | Lee Yarwood proposed openstack/nova master: api: Introduce microverion 2.82 allowing boot from volume rescue https://review.opendev.org/701430 | |
| 14:50:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Support boot from volume stable device instance rescue https://review.opendev.org/701431 | |
| 14:50:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Extract _get_bdm_image_metadata into nova.utils https://review.opendev.org/705212 | |
| 15:01:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - Test stable device rescue tests with BFV instances https://review.opendev.org/710050 | |
| 16:06:35 | efried | sean-k-mooney: Re token expiring on that cyborg job: is the conf set up with a service token? | |
| 16:12:22 | sean-k-mooney | i am not sure but the confs are in the job logs | |
| 16:12:28 | sean-k-mooney | it should not fail in either case | |
| 16:12:47 | sean-k-mooney | once it start happening you have to restart all the nova services to fix it | |
| 16:22:50 | sean-k-mooney | efried: so no https://48ef08cde8cc22034a1d-8011a2266d21f0c09baf1c83d6d5002e.ssl.cf5.rackcdn.com/709641/5/check/cyborg-multinode-tempest-full/e4d260f/controller/logs/etc/nova/nova_conf.txt | |
| 16:23:12 | sean-k-mooney | the service user section https://docs.openstack.org/nova/latest/configuration/config.html#service-user | |
| 16:24:14 | sean-k-mooney | is not configured but it should not need to be confitured for it to work | |
| 16:24:44 | sean-k-mooney | it might allow you to mask the issue | |
| 16:27:35 | efried | sean-k-mooney: so the failure is happening from a later operation, not from the middle of a long-running operation? | |
| 16:46:05 | gmann | brinzhang__: sure, sorry for late response. I will review your patches today | |
| 16:50:40 | efried | sean-k-mooney: can I get your nod on https://review.opendev.org/#/c/709902/ (rocky EM patch) for os-vif please? | |
| 17:06:32 | efried | lyarwood: https://review.opendev.org/#/c/709902/1/deliverables/rocky/nova.yaml lgty? | |
| 17:12:08 | lyarwood | efried: ack yes, elod ^? | |
| 17:33:03 | sean-k-mooney | efried: it can start failing with a 401 in the middel of an operation and once it has it will continue to fail for seperate operations | |
| 17:33:41 | sean-k-mooney | and yes ill look at the em patch now | |
| 17:35:28 | sean-k-mooney | there is one pending bugfix i want to back port to all affected branch in os-vif but rocky predates the issue so yes that commit looks correct | |
| 17:35:32 | sean-k-mooney | ill +1 the review | |
| 17:38:23 | sean-k-mooney | efried: basicaly the way i first hit the cybrog issue was i booted a vm. did a bunch of life cycle operation on it and then tried to delete it and that failed befaue the token was rejected. | |
| 17:38:36 | sean-k-mooney | when i tried to do the operation (deleting the arq) myself it worked | |
| 17:39:25 | sean-k-mooney | if i listed device profiles myslef it also worked but when i tried to boot anotuher vm after that point it failed beacuse the nova api was not able to retive the device profile info | |
| 17:40:44 | sean-k-mooney | so if i use osc to query cyborg directly everthing is fine. if i use osc to boot a vm and nova tires to query cyborg on my behalf it was failing but only after the services had been running for a while like an hour or so | |
| 18:24:17 | lucidguy | Anyone recall me asking for assistance with >1tb ram instances? I FIGURED IT OUT!!! only took days. | |
| 18:25:23 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/708745 | |
| 18:27:10 | lyarwood | lucidguy: what was it? | |
| 18:27:11 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Provide the backing file format when creating qcow2 disks https://review.opendev.org/708745 | |
| 18:32:44 | sean-k-mooney | updating other people code when you have never reviewed it and are just following gerrit comments is hard | |
| 18:33:19 | sean-k-mooney | the changes arent hard but the mental load to make sure what you are doing is correct is way higher then when its your code | |
| 18:39:37 | melwitt | lucidguy: don't leave us hanging | |
| 18:44:30 | sean-k-mooney | lucidguy: was it the alingment of jupiture | |
| 18:44:50 | sean-k-mooney | *jupiter | |
| 19:15:22 | openstackgerrit | Merged openstack/python-novaclient master: Don't print user_data for 'nova show' https://review.opendev.org/708850 | |
| 19:51:16 | lucidguy | sean-k-mooney? | |
| 19:51:19 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - Test stable device rescue tests with BFV instances https://review.opendev.org/710050 | |
| 19:53:41 | lucidguy | sean-k-mooney: By default instances are launched with 40bit cpu memory address space, that does not allow for >1tb memory in an instances. qemu on Ubuntu 18.04 allow to choose a machine architecture that maps the instances address space with the local HV which is 46bits. In the end of the day all I had to do is upgrade to 18.04(Bionic) and add one line to nova.conf on the HV. | |
| 20:08:10 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 20:20:20 | sean-k-mooney | lucidguy: ah yes that makes sense | |