| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-18 | |||
| 03:17:36 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [Do not Merge] test Arm64 CI https://review.opendev.org/708294 | |
| 04:39:50 | openstackgerrit | Merged openstack/nova stable/pike: Use stable constraint for Tempest pinned stable branches https://review.opendev.org/706715 | |
| 06:52:22 | openstackgerrit | Brin Zhang proposed openstack/nova master: FUP Populate the 'updated_at' when aggregate metadata is updated https://review.opendev.org/702790 | |
| 07:10:57 | alex_xu | rosmaita: I think it needn't a microversion | |
| 07:41:14 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] Add arm64 CI https://review.opendev.org/708102 | |
| 07:48:42 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] Add arm64 CI https://review.opendev.org/708102 | |
| 07:50:18 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [Do not Merge] test Arm64 CI https://review.opendev.org/708293 | |
| 07:50:46 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [Do not Merge] test Arm64 CI https://review.opendev.org/708294 | |
| 09:04:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unnecessary exception handling, mocks https://review.opendev.org/703973 | |
| 09:04:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 09:04:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974 | |
| 09:39:24 | lyarwood | efried: sorry was afk yesterday sick, https://review.opendev.org/#/q/topic:spec/virt-rescue-stable-disk-devices needs to land ahead of https://review.opendev.org/#/q/topic:bp/virt-bfv-instance-rescue - working on the top WIP libvirt change today, it's mostly unit/tempest coverage left to do at this point. | |
| 09:43:36 | kashyap | efried: On that secure boot spec, it's been on the back-burner as I was buried in other stuff, but I definitely want to get it done. | |
| 10:14:01 | brinzhang_ | efried, gibi, stephenfin, alex_xu: Do we have to follow nova runway, it has not been updated for 5 days :). https://etherpad.openstack.org/p/nova-runways-ussuri | |
| 10:20:09 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180 | |
| 10:39:17 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 10:39:19 | openstackgerrit | Ilya Etingof proposed openstack/nova master: Add JSON schema and test for network_data.json https://review.opendev.org/703133 | |
| 10:39:42 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180 | |
| 10:55:55 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/708340 | |
| 11:26:32 | gibi | brinzhang_: good question. let me update the runways | |
| 11:26:50 | sean-k-mooney | efried: dansmith trying to do multi node im hitting some db errors with arq binding http://paste.openstack.org/show/789695/ im going to take a look at the db and see if i can figure out what is going on quickly but at present | |
| 11:26:57 | sean-k-mooney | i cant spawn a vm | |
| 11:27:12 | sean-k-mooney | gibi brinzhang_ we have not really been using runways this cycle | |
| 11:27:32 | sean-k-mooney | not that im against using them we just have not been using them for some reason | |
| 11:34:47 | gibi | brinzhang_, efried, gmann, lyarwood, stephenfin: I've update the runways etherpad | |
| 11:35:21 | gibi | sean-k-mooney: yeah. I've added a topic for the PTG about runways | |
| 11:36:08 | gibi | sean-k-mooney: did cyborg updated the fake driver to report the RP with a unique name? months ago when I looked at it my problem was that both compute node used the same RP name for the fake FPGA resource | |
| 12:14:46 | sean-k-mooney | ah right that could be the issue | |
| 12:14:51 | sean-k-mooney | let me check | |
| 12:15:15 | sean-k-mooney | gibi: also i responded to some of your questions in https://review.opendev.org/#/c/673735/38 | |
| 12:16:36 | sean-k-mooney | gibi: ya that is the issue | |
| 12:16:40 | sean-k-mooney | ther is only 1 rp | |
| 12:17:08 | sean-k-mooney | well there are 3 1 for each compute and then 1 for the fake device | |
| 12:17:17 | sean-k-mooney | so the fake driver is incomplete | |
| 12:18:39 | sean-k-mooney | i also noticed the devstack plugin is incorrect. it does not properly support deploy just the comptue agent like it sugges. it will not create the correct cybog config so the agent will be installed but wont be able to connect to rabbitmq | |
| 12:19:30 | sean-k-mooney | dansmith: so based on the fact that the RP is hardcoded im going to say creating a tempory multi node job will be more work then i had guessed | |
| 12:19:57 | sean-k-mooney | it could still be done but i would have to modify the cyborg code to fix the devstack plugin and correct the hardcoding | |
| 12:24:33 | gibi | sean-k-mooney: ack thanks, I will get back to that patch after I finished with the whole sereies | |
| 12:24:36 | gibi | series | |
| 12:34:17 | openstackgerrit | Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811 | |
| 12:34:17 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812 | |
| 12:36:46 | gibi | efried, dansmith: I finished reading the cyborg series. The re-schedule is not covered with functional test so I'm not sure it works. I also found other missing test coverage as well | |
| 12:38:15 | rosmaita | alex_xu: thank you | |
| 12:40:21 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815 | |
| 12:41:07 | sean-k-mooney | gibi: looking at the fake cyborg driver i dont imediatly see where they are setting the RP name meaning they my be hardcoding it above the dirver meaning im not sure you coudl ever have more then one cyborg rp. ill keep looking but has anyone tested multinode with 2 hosts that have fpgas? | |
| 12:41:46 | openstackgerrit | Luyao Zhong proposed openstack/nova master: bug-fix: Reject live migration with vpmem https://review.opendev.org/708110 | |
| 12:41:47 | openstackgerrit | Luyao Zhong proposed openstack/nova master: address specific resources cleanup issue https://review.opendev.org/699148 | |
| 12:41:47 | openstackgerrit | Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856 | |
| 12:41:57 | gibi | sean-k-mooney: thanks for the answers in https://review.opendev.org/#/c/673735 much appreciated | |
| 12:42:22 | gibi | sean-k-mooney: I have to reload the context but months ago I was able to found the place where the hard-coding was done | |
| 12:42:44 | sean-k-mooney | the repo is small so it wont take me long to find it | |
| 12:44:27 | sean-k-mooney | i dont think i want to know why the cyborg agent calls ethe cyborg conductor to diff ther ereport data as part of updating the inventory do i... | |
| 12:44:52 | sean-k-mooney | agents shoudl really not do that | |
| 12:45:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815 | |
| 12:48:35 | sean-k-mooney | gibi: ... so cyborg are doing all the inventory updates via the conductor by haveing the agents inventory the local host then doign an rpc to the conductor which does the diff of the old state to the new state then it updates the placement invtories | |
| 12:49:12 | sean-k-mooney | that is not goign to scale | |
| 12:50:57 | sean-k-mooney | also why are they using the dns_namespace for https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/conductor/manager.py#L417-L418 | |
| 12:52:14 | gibi | sean-k-mooney: most probably they want to have stable uuids | |
| 12:52:36 | gibi | but I don't know why the DNS namespace is used for that | |
| 12:52:39 | sean-k-mooney | yes but they should have defiend there own namespace also the shoudl be using uuid5 not uuid3 | |
| 12:52:58 | gibi | sean-k-mooney: agree | |
| 12:53:11 | gibi | uuid5 with a hardcoded namespace would be good | |
| 12:53:19 | sean-k-mooney | yep | |
| 12:53:31 | sean-k-mooney | md5 is old an busted and not fips compliant | |
| 12:54:08 | sean-k-mooney | uuid5 uses sha1 which is slightly less old, is fips complant and has hardwar instruction in the cpu | |
| 12:57:11 | gibi | honestly I lost in the code when I wanted to track down the RP naming | |
| 12:57:49 | gibi | the code change significantly since last time I looked into it | |
| 12:57:49 | sean-k-mooney | the painful thing is to change it you basicaly now need to do a cuckoo hash implemation to fall back to the old way of generating the uuid if you did not find it | |
| 12:58:36 | sean-k-mooney | gibi: ya i think the code will have to be refactored on there end before it can go into production as there is no way doing all the updated form the conductor like that will work for large deployments | |
| 12:59:25 | sean-k-mooney | the current architure will ddos the cyborg conductor with the periodic task updates | |
| 13:09:34 | sean-k-mooney | ah i found it | |
| 13:10:19 | gibi | sean-k-mooney: ohh, link it to me please :) | |
| 13:10:25 | sean-k-mooney | https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L70 | |
| 13:10:51 | sean-k-mooney | when they generate the deployable list https://github.com/openstack/cyborg/blob/369abe8dd06aa6648298c3256f444a63ee6268d0/cyborg/accelerator/drivers/fake.py#L59 | |
| 13:10:54 | gibi | sean-k-mooney: thanks. | |
| 13:11:00 | sean-k-mooney | they use the device key form the dict | |
| 13:11:08 | gibi | yepp, I saw that | |
| 13:11:08 | sean-k-mooney | at least in the fake driver | |
| 13:11:28 | gibi | basically they need to include the host name when they convert the device name to RP name | |
| 13:11:36 | sean-k-mooney | and the conductor uses the name to get the uuid form the deployable | |
| 13:11:42 | sean-k-mooney | yep | |
| 13:11:49 | sean-k-mooney | im going to hack that in locally | |
| 13:13:38 | sean-k-mooney | basically driver_dep.name = "%s_%s" % (hostname,pci.get('device')) | |
| 13:13:53 | gibi | yep | |
| 13:14:12 | sean-k-mooney | although i might use the conf.host vlaue instead of the hostname | |
| 13:14:32 | sean-k-mooney | there is a utility function i think for this already | |
| 13:29:39 | gmann | thanks gibi | |
| 13:30:16 | sean-k-mooney | gibi: ok cool that appears to fix the placement issue but im still geting keystone auth issues | |
| 13:31:17 | sean-k-mooney | spcifically devstack@cyborg-api.service[8430]: 2020-02-18 13:28:51.849 8431 WARNING keystonemiddleware.auth_token [req-640319ce-6b48-48f5-a406-9038b51d0e67 cb8c1a4ddbc84fc5aadf961423c0a843 8ad0765aa5f84400b394ee05794f6402 - default default] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed | |
| 13:32:11 | sean-k-mooney | the token im sendign form the client is valid as i can do server create/delete so either the nova code is passing an expired token or more likely the cyborg api is not handeling it correctly | |
| 13:36:57 | brinzhang_ | gibi, sean-k-mooney: I have seen, thanks for update that, I think it's useful for some want to review some working patches. | |
| 13:38:25 | sean-k-mooney | ok so if i restart the nova service then the auth issue goes away i think but i think get other errors | |
| 13:38:28 | brinzhang_ | s/some/someone | |
| 13:38:47 | sean-k-mooney | on the plus side it did reschduler but i hav enot read through the logs to see if it did the right thing | |
| 13:43:23 | sean-k-mooney | gibi: dansmith efried those are the logs for the failed boot that tried to resudule. http://paste.openstack.org/show/789700/ | |
| 13:43:43 | sean-k-mooney | i need to go offline for a few hours i will be back online this evening for an hour or so | |
| 13:46:36 | sean-k-mooney | the instance uuid is 4cefbecb-7ed1-4331-84d9-03f8d13f3ef6 but there looks like ther there is an error in how the compute mnager loops over the arqs and ther emight be other issues | |
| 13:55:38 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180 | |
| 13:58:11 | openstackgerrit | John Garbutt proposed openstack/nova master: DNM: Turn on unified limits to see if tests pass https://review.opendev.org/708387 | |
| 14:22:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova.image.api' module https://review.opendev.org/702451 | |