| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 12:51:02 | efried | sean-k-mooney: https://review.openstack.org/#/c/561677/ <== rocky. So yeah. | |
| 12:51:05 | sean-k-mooney | i think that is correct | |
| 12:51:24 | shilpasd | efried: sean-k-mooney: okay, will check with stephenfin, 'instances_path = /opt/stack/data/nova/instances' | |
| 12:51:49 | openstackgerrit | Matthew Booth proposed openstack/nova master: systemd detection result caching nit fixes https://review.openstack.org/649229 | |
| 12:51:50 | sean-k-mooney | efried: so ya not an RC candiate but assuming the fix is non invasive it should be a backprot candiate | |
| 12:52:09 | sean-k-mooney | shilpasd: is this a devstack setup | |
| 12:53:33 | shilpasd | sean-k-mooney: yes | |
| 12:53:43 | sean-k-mooney | if so the /opt/stack/data directory is where we sotre image and cinder voluems and other perstnet data form the openstack services | |
| 12:53:56 | sean-k-mooney | so i would not be surpiesd if the instacne root disk is also stored there | |
| 12:54:20 | sean-k-mooney | i have never really checked however. | |
| 12:54:41 | shilpasd | sean-k-mooney: but in multinode case, it was stored at both node in my case, at controller side and at compute side | |
| 12:55:08 | sean-k-mooney | shilpasd: it wont be sotre in both localtions it will be copied if you do a migration | |
| 12:55:42 | sean-k-mooney | e.g. we wont create the isntace root disk on all nodes in the multinode devstack cloud | |
| 12:56:03 | openstackgerrit | Eric Fried proposed openstack/nova stable/stein: Adding tests to demonstrate bug #1821824 https://review.openstack.org/649600 | |
| 12:56:04 | openstackgerrit | Eric Fried proposed openstack/nova stable/stein: Fix bug preventing forbidden traits from working https://review.openstack.org/649601 | |
| 12:56:04 | shilpasd | sean-k-mooney: yes, but before migration i am verifying 'instances_path' and observed that instance files are already there | |
| 12:56:04 | openstack | bug 1821824 in OpenStack Compute (nova) stein "Forbidden traits in flavor properties don't work" [High,Confirmed] https://launchpad.net/bugs/1821824 | |
| 12:56:30 | shilpasd | sean-k-mooney: IMO it should not be the case, as you said during migration it should copy | |
| 12:56:56 | sean-k-mooney | if you are doing a block migration it will copy them yes | |
| 12:57:35 | sean-k-mooney | you can also mount /opt/stack/data/nova on nfs to avoid this but you would have to do that manually | |
| 12:57:38 | openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Adding tests to demonstrate bug #1821824 https://review.openstack.org/649602 | |
| 12:57:39 | openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603 | |
| 12:57:39 | openstack | bug 1821824 in OpenStack Compute (nova) stein "Forbidden traits in flavor properties don't work" [High,In progress] https://launchpad.net/bugs/1821824 - Assigned to Eric Fried (efried) | |
| 12:58:01 | sean-k-mooney | shilpasd: are you tryign to debug somthing locally or in the gate | |
| 12:58:17 | sean-k-mooney | or just understand how it works | |
| 12:58:23 | shilpasd | sean-k-mooney: tried with NFS + multinode, there saw insatnce data file created at all 3 places in my case, at NFS shared location, at controller and at compute node. | |
| 12:58:42 | sean-k-mooney | ah its not create at all 3 locations | |
| 12:59:06 | sean-k-mooney | the data is stored on the nfs share and its just mounted on the contoler and compute | |
| 12:59:24 | sean-k-mooney | there is one copy of the data but its acceable via nfs form multiple locations | |
| 12:59:36 | shilpasd | sean-k-mooney: yes me to wondered, and here during evacuation getting error 'libvirtError: internal error: process exited while connecting to monitor.', 'ERROR oslo_messaging.rpc.server Is another process using the image?' | |
| 13:00:10 | sean-k-mooney | shilpasd: what version of openstack are you deploying | |
| 13:00:46 | shilpasd | sean-k-mooney: stein | |
| 13:01:02 | sean-k-mooney | when we do an evacuate we assume the guest on the donw node is stoped | |
| 13:01:09 | sean-k-mooney | if its not its not safe to evacuate | |
| 13:01:14 | shilpasd | openstack 3.18.0 | |
| 13:01:23 | sean-k-mooney | did you stop the vm? | |
| 13:01:44 | shilpasd | purposefully stopping n-cpu by service-force-down | |
| 13:02:00 | sean-k-mooney | that will not stop the vm | |
| 13:02:27 | shilpasd | yes but evacuation happens in this case also | |
| 13:02:57 | sean-k-mooney | yes and the if you use force down you are required to check that the vms are stopped on the host | |
| 13:03:45 | sean-k-mooney | force does was added specficaly for operator that had external monitorin that could determin the host had failed and would stop all reunning guest before seting it | |
| 13:04:19 | shilpasd | ok, will check that, but IMO error i am getting is because of same data files already available and during evacuation the said error ''ERROR oslo_messaging.rpc.server Is another process using the image?'' | |
| 13:04:33 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html#use-cases | |
| 13:05:02 | shilpasd | ok, will check after VM down, but original question remains, why insatnce data files created at all 3 places NFS, controller and compute? | |
| 13:05:16 | shilpasd | is it correct behaviour? | |
| 13:05:22 | sean-k-mooney | yes | |
| 13:05:34 | sean-k-mooney | assumign that the instance directory is on nfs | |
| 13:05:53 | sean-k-mooney | what does mount show on the contoler or compute node | |
| 13:06:32 | sean-k-mooney | we have special logic that detect if the instace directory is on nfs and skips copying the disk in that cases | |
| 13:06:33 | shilpasd | same shared location | |
| 13:06:55 | shilpasd | can you please help me where exatly this logic is | |
| 13:07:23 | shilpasd | i have checked def _create_image() of libvirt | |
| 13:08:26 | openstackgerrit | Jared Winborne proposed openstack/nova master: Leave brackets on Ceph IP addresses for libguestfs https://review.openstack.org/649405 | |
| 13:10:08 | sean-k-mooney | shilpasd: https://github.com/openstack/nova/blob/1554d35834a474514f827449bd7d4f1d2f0af1d6/nova/virt/libvirt/driver.py#L6611-L6641 | |
| 13:11:19 | sean-k-mooney | shilpasd: the issue here however is not related to this check. | |
| 13:11:36 | KH-Jared | I'm just going to blame my long lines on trusting my IDE too much. Its pep8 warning was at 120 characters instead of 79, whoops | |
| 13:12:07 | sean-k-mooney | if you use force host down, you are required to ensure the vm is not running before you call evacuate. that is the cause of the error you hit | |
| 13:13:20 | shilpasd | sean-k-mooney: yes, thanks for this input, will check with VM down, and will debug more the given refrence code, and get back to you on the same later | |
| 13:13:25 | sean-k-mooney | KH-Jared: ya the 79 column limmit is particlaly annoying because no ide defualt to 79. some defult to 80 but even pycharm does not defualt to 79 and its a python focused ide | |
| 13:15:15 | KH-Jared | I can get behind it though, the code has always been extremely nice to read, I assume it adds to that. I also have it fixed in my ide now, so hopefully won't see a pep8 failure again in the future | |
| 13:16:08 | sean-k-mooney | i have been working on openstack for the better part of 6 years and i still dont write pep8 complient code by default | |
| 13:16:26 | sean-k-mooney | but i run fast8 on it most of the time before i push | |
| 13:16:44 | sean-k-mooney | KH-Jared: you are aware of the fast8 env | |
| 13:16:50 | sean-k-mooney | tox -e fast8 | |
| 13:16:53 | KH-Jared | i am now | |
| 13:17:08 | sean-k-mooney | it runs pep8 but just on your patched files | |
| 13:17:13 | sean-k-mooney | not on all of nova | |
| 13:17:15 | sean-k-mooney | is way fater | |
| 13:17:17 | artom | sean-k-mooney, whoa, that exists? | |
| 13:17:21 | sean-k-mooney | *faster | |
| 13:17:30 | sean-k-mooney | artom: yes... | |
| 13:17:43 | artom | I've been hacking something like it with $(tox -e pep8 `git show --name-only | grep ^nova`) | |
| 13:18:09 | sean-k-mooney | artom: stephenfin added it in pike | |
| 13:18:16 | artom | ... | |
| 13:18:20 | KH-Jared | this was my first change on a large project, not even just openstack, all testing I've done has been much smaller by comparison. I'm going to be using flake8 more often atleast | |
| 13:18:21 | artom | \o/ | |
| 13:18:22 | sean-k-mooney | its also in a few other repos at this point | |
| 13:18:44 | efried | jaypipes: Do we still have one RT per ironic node? | |
| 13:19:23 | mriedem | gibi_off: do you know why this was made into a warning? https://github.com/openstack/nova/blob/b33fa1c054ba4b7d4e789aa51250ad5c8325da2d/nova/scheduler/client/report.py#L1880 we hit that a lot in normal resizes: https://bugs.launchpad.net/nova/+bug/1822917 | |
| 13:19:24 | openstack | Launchpad bug 1822917 in OpenStack Compute (nova) ""Overwriting current allocation" warnings in logs during move operations although there are no failures" [Undecided,In progress] - Assigned to Takashi NATSUME (natsume-takashi) | |
| 13:20:42 | mriedem | efried: there is one RT per nova-compute service, | |
| 13:20:47 | mriedem | and the RT has a dict of compute nodes | |
| 13:21:13 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L544 | |
| 13:21:32 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L139 | |
| 13:21:52 | efried | mriedem: I'm looking at stephenfin's change https://review.openstack.org/#/c/649559/ which seems sane in itself, but https://review.openstack.org/#/c/649559/1/nova/compute/resource_tracker.py@605 is only being run through once, for the "first" node. | |
| 13:22:26 | efried | it's probably moot by luck because we don't track PCI devices on ironic nodes (right?) | |
| 13:23:20 | mriedem | probably | |
| 13:25:29 | jaypipes | efried: no, not for years. | |
| 13:25:44 | efried | swhat I thought, see above | |
| 13:26:22 | jaypipes | k, will look. | |
| 13:29:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead resource tracker code https://review.openstack.org/649569 | |
| 13:30:58 | sean-k-mooney | stephenfin: regarding ^ this conflits with some other chages that might be starting to used some of that dead code | |
| 13:31:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead nova.db functions https://review.openstack.org/649570 | |
| 13:31:30 | sean-k-mooney | im thinking about https://review.openstack.org/#/q/topic:bug/1809095+(status:open+OR+status:merged) | |
| 13:32:19 | stephenfin | sean-k-mooney: Possibly. If so, let me know. I was just using vulture (https://pypi.org/project/vulture/) to figure that stuff out so there's a lot of missing context | |
| 13:32:25 | sean-k-mooney | stephenfin: actully no never mind | |
| 13:32:27 | sean-k-mooney | they are not | |
| 13:32:35 | sean-k-mooney | its just a conflit in the tests | |
| 13:33:16 | sean-k-mooney | stephenfin: we might have some function that were added for the sriov migration code too | |
| 13:33:31 | sean-k-mooney | we merged all the resouce tracker code but have not merged the 2 patches that use it | |