| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 11:48:53 | openstackgerrit | John Garbutt proposed openstack/nova master: Stop failed live-migrates getting stuck migrating https://review.openstack.org/430404 | |
| 11:55:34 | johnthetubaguy | sfinucan: if you get chance, this is another nasty live-migrate bug that was blocking the live-migration performance testing OSIC are doing: https://bugs.launchpad.net/nova/+bug/1662626 | |
| 11:55:34 | openstack | Launchpad bug 1662626 in OpenStack Compute (nova) "live-migrate left in migrating as domain not found" [Medium,In progress] - Assigned to John Garbutt (johngarbutt) | |
| 11:56:08 | johnthetubaguy | sfinucan: along with this one: https://review.openstack.org/#/c/429798/5 | |
| 11:56:12 | sfinucan | johnthetubaguy: so https://review.openstack.org/#/c/430404/ ? | |
| 11:56:27 | johnthetubaguy | sfinucan: both of them | |
| 11:56:34 | johnthetubaguy | well, all three I guess | |
| 11:56:34 | sfinucan | gotcha | |
| 11:56:57 | johnthetubaguy | one fixes the root cause of the error, the other fixes our handing of the error, if that makes sense | |
| 11:57:12 | sfinucan | It does | |
| 11:57:19 | sfinucan | I'll take a look shortly, once I'm done with lyarwood / mdbooth's two fixes | |
| 12:01:00 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance-exists notification https://review.openstack.org/403660 | |
| 12:04:31 | lyarwood | sfinucan: we should start a club | |
| 12:05:24 | lyarwood | or a therapy group | |
| 12:20:55 | mdbooth | lyarwood: Awesome, thanks for doing that. | |
| 12:21:05 | mdbooth | lyarwood: Just making the log level change now. | |
| 12:21:27 | openstackgerrit | MarkMielke proposed openstack/nova master: DiskFilter should only apply when storage is requested https://review.openstack.org/430091 | |
| 12:22:36 | lyarwood | mdbooth: np | |
| 12:22:38 | lyarwood | gerrit-- | |
| 12:23:36 | mdbooth | lyarwood: Incidentally, the patch I posted for the detach race doesn't work, because the race isn't local to the compute host | |
| 12:23:49 | mdbooth | Which is annoying | |
| 12:28:09 | lyarwood | mdbooth: wonderful, so I guess we just remove the detach from the compute layer, see which virt drivers we break and add in an additional detach there? | |
| 12:28:56 | mdbooth | lyarwood: The issue is that the bdm list is fetch on the api node and passed to the compute node, but is out of date by the time it's used. | |
| 12:29:21 | mdbooth | So we can take a lock on the compute node, but we're still operating on a list which was generated outside of that lock. | |
| 12:29:53 | mdbooth | It would also race with anything else which modified bdms other than the compute node | |
| 12:31:39 | lyarwood | mdbooth: isn't the compute the only thing modifying bdms for running instances? | |
| 12:32:04 | mdbooth | lyarwood: I doubt it, tbh. | |
| 12:32:32 | mdbooth | I wouldn't be surprised to discover that api and conductor also touch them at various times. | |
| 12:32:44 | mdbooth | No specific examples come to mind, just my expectation. | |
| 12:33:30 | openstackgerrit | Matthew Booth proposed openstack/nova master: libvirt: Fix races with nfs volume mount/umount https://review.openstack.org/383859 | |
| 12:33:51 | mdbooth | lyarwood sfinucan johnthetubaguy: ^^^ no changes other than some log levels | |
| 12:42:33 | sfinucan | lyarwood: Finally worked my way though https://review.openstack.org/#/c/427364/ | |
| 12:42:51 | sfinucan | up to you to someone else for the +W though :) | |
| 12:43:01 | sfinucan | /-1 | |
| 12:44:49 | alex_xu | nova api meeting is in 15 mins at #openstack-meeting-4 | |
| 12:53:59 | openstackgerrit | John Garbutt proposed openstack/nova master: Remove stuck live-migration detection https://review.openstack.org/430218 | |
| 12:54:00 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Avoid all live-migration timeouts by default https://review.openstack.org/430890 | |
| 12:57:15 | lyarwood | sfinucan: ah awesome thanks! I'll chase another core review once the tempest test is passing, appears to be hitting a few unrelated issues atm | |
| 12:59:53 | sfinucan | How on earth did no one detect https://bugs.launchpad.net/nova/+bug/1644248 before now? | |
| 12:59:53 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "Nova incorrectly tracks live migration progress" [High,In progress] - Assigned to John Garbutt (johngarbutt) | |
| 13:00:57 | alex_xu | johnthetubaguy: sdague are you around for nova api meeting today? plan to talk about Pike plan today | |
| 13:02:59 | sfinucan | johnthetubaguy: I think you need a deprecation reno section in https://review.openstack.org/#/c/429798/5 | |
| 13:10:43 | sfinucan | johnthetubaguy, mdbooth: Think I've got all those patches reviewed. Now for foooood | |
| 13:23:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'ipv6_backend' https://review.openstack.org/430716 | |
| 13:23:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Fix indentation in conf/netconf https://review.openstack.org/430717 | |
| 13:23:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'firewall_driver' https://review.openstack.org/412390 | |
| 13:23:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Ignore 'use_ipv6' for port filters https://review.openstack.org/430909 | |
| 13:23:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: netutils: Ignore 'use_ipv6' for network templates https://review.openstack.org/430910 | |
| 13:23:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'use_ipv6' https://review.openstack.org/430911 | |
| 13:23:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Ignore 'allow_same_net_traffic' for port filters https://review.openstack.org/430912 | |
| 13:23:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'allow_same_net_traffic' https://review.openstack.org/430913 | |
| 13:23:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'metadata_host', 'metadata_port' https://review.openstack.org/430914 | |
| 13:23:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate various 'iptables_' options https://review.openstack.org/430915 | |
| 13:23:59 | sfinucan | johnthetubaguy: ^ fixed the use_ipv6 use case, and then some :) | |
| 13:28:42 | lpetrut | Hi, I have a question regarding cells caching: did anyone propose a patch already, refreshing n-api's cached cells list? https://github.com/openstack/nova/blob/466769e588dc44d11987430b54ca1bd7188abffb/nova/compute/api.py#L109-L112 | |
| 13:56:09 | johnthetubaguy | lpetrut: I don't think so, because we only support having one cell right now (other than for testing) | |
| 13:57:06 | lpetrut | johnthetubaguy: got it, thanks | |
| 13:58:52 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse virt_type attribute of LibvirtConfigGuest from xml https://review.openstack.org/414747 | |
| 13:58:53 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse os attributes of LibvirtConfigGuest from xml https://review.openstack.org/410665 | |
| 13:58:53 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse filesystem elements of guest config https://review.openstack.org/410666 | |
| 13:58:54 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Use config types to parse XML for instance disks https://review.openstack.org/410667 | |
| 13:58:54 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Add missing tests for utils.find_disk https://review.openstack.org/414750 | |
| 13:58:55 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse basic attributes of LibvirtConfigGuest from xml https://review.openstack.org/414748 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Use config types to parse XML for root disk https://review.openstack.org/411941 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Introduce Guest.get_config method https://review.openstack.org/414749 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Simplify internal usage of get_instance_disk_info https://review.openstack.org/334901 | |
| 13:58:57 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Remove unused disk_info parameter https://review.openstack.org/382025 | |
| 14:00:34 | mordred | sdague: thanks for reminding me I need to implement the needs-floating-ip flag | |
| 14:01:05 | mordred | sdague: it's one of the best examples, because discovering whether the cloud requires a floating ip or not involves _several_ api calls and a bunch of guess work | |
| 14:02:43 | sdague | mordred: yeh, so it feels like we should flip this whole thing over and start with the shade view of the world and how we'd effectively expose those things to you | |
| 14:05:02 | mordred | sdague: yah. I think that's likely a good and concrete starting point | |
| 14:05:45 | EmilienM | cdent: hey, I need your help if you have time | |
| 14:05:53 | mordred | sdague: re: putting stuff in the catalog, we have a cloud vendor profile data structure in os-client-config that might be an 'easy' thing to put into the catalog as a starting point | |
| 14:06:04 | cdent | EmilienM: I do, but I need to get some coffee first. cool? | |
| 14:06:08 | EmilienM | cdent: sure thing | |
| 14:06:16 | EmilienM | cdent: I'll paste things here | |
| 14:06:22 | mordred | (it needs a little clean up and more docs of course) | |
| 14:06:39 | EmilienM | so I'm deploying Nova from trunk, in eventlet (we removed WSGI/Apache for nova API like you suggested) | |
| 14:07:17 | EmilienM | we start nova services and then run the discover hosts command. Though it seems like the nova scheduler can't find the compute node registered | |
| 14:07:33 | EmilienM | and it looks like the compute didn't register to placement api: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:07:40 | mordred | sdague: so like instead of this: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/vendors/rackspace.json being in the os-client-config repo, it could just be in the catalog (which, of course, is mildly problematic because auth_url and auth_type still need to be discovered from somewhere, but baby steps) | |
| 14:08:26 | EmilienM | you can see nova-conductor which fails to find compute resource: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-08_06_41_56_037 | |
| 14:11:50 | johnthetubaguy | EmilienM: the nova-compute will need to be registered in its compute database before you run the discover_hosts cmd, it could be a bit racey, depending on when its called I guess? | |
| 14:12:28 | EmilienM | johnthetubaguy: so that's exactly what we were looking at | |
| 14:12:38 | EmilienM | johnthetubaguy: let me share timestamps with you | |
| 14:13:05 | EmilienM | at 06:38:25 we run nova-cell_v2-discover_hosts | |
| 14:13:21 | EmilienM | but we start nova compute at 06:32:06 | |
| 14:13:31 | johnthetubaguy | EmilienM: I think the three is being registered in the cell, registered in placement and registered in the API | |
| 14:13:46 | johnthetubaguy | let me find the compute log, one sec... | |
| 14:13:49 | EmilienM | we think nova compute is started at http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_19_476 | |
| 14:14:28 | johnthetubaguy | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_30_59_875 | |
| 14:14:56 | johnthetubaguy | thats where you see the compute node being created, i.e. registered in the child cell | |
| 14:15:01 | johnthetubaguy | you need to discover hosts after that | |
| 14:15:06 | johnthetubaguy | I think | |
| 14:16:14 | johnthetubaguy | hmm, I guess I don't understand the state of the system | |
| 14:16:28 | johnthetubaguy | it looks more like placement is not returning any hosts, I could be wrong | |
| 14:16:48 | owalsh | johnthetubaguy: I don't see any POST requests in the placement api log | |
| 14:17:07 | cdent | EmilienM, johnthetubaguy: I'm back, catching up. which is the best log to start with? | |
| 14:17:36 | johnthetubaguy | owalsh: yeah, compute logs say its not configured in the compute log, at least in the one I am looking at | |
| 14:17:53 | johnthetubaguy | owalsh: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_01_297 | |